Playlist being created with issues

[Augusto7743]

Using Retroarch 1.9.11 in Linux Ubuntu 20.04.
When creating a playlist for Atari 2600 Stella core was selected a folder with the files named below
Demos.zip
Educational.zip
Programs.zip
ROMs 1.zip
ROMs 2.zip
ROMs 3.zip
ROMs 4.zip

Playlist creation to search inside files is enabled.
The playlist is created with issues only allowing start few games.

Not issues with zip files because open correctly in Stella.
Unhappily not is possible create a playlist to point to a zip file and browse inside to select a rom.

[i30817]

Retroarch so called 'automatic scanner' (i call it 'useless scanner') doesn't treat zips as directories. It treats them as 'single rom containers'. Single games in a single zip basically.

And when that doesn't work, or isn't in the database, it has a hell of time identifying games or uses methods prone to false negatives or laborious workarounds (lol Scummvm and DOS).

This, as you just found out, means that the first 'file with the correct extension for one of cores available' is assigned as the 'rom' of a zip, even if that zip contains multiple such files (for the automatic scanner it also has to match a checksum on the rom database).

Last time i checked (might have been 'fixed' in the meanwhile) this was especially funny when the zip was a compressed cd with multiple tracks or mp3 tracks or cue/bins because retroarch didn't have any problems just extracting the largest .bin it could find and loading just that, breaking many of redump games (no cd sound, wrong binary mode, etc).

The one exception to this, is the mame/fbneo romset, where the zip themselves are the 'rom' if the zips are zipped with special zip tool that creates a deterministic zip (so the checksums match the merged/split sets).

Do yourself a favor and use the manual scanner, maybe with 'scan inside archives' (i never tried this option but it might work for this).

[i30817]

Also be aware that retroarch is simply terrible for disk wear at handling compressed content if you're not explicitly using a ramdisk for temporary files.

You don't 'stream' a file from the compressed file and into memory. You 'uncompress' a file and then give that to the core. Which means that if your zip is huge, you're uncompressing a huge thing every time you load content.

The exception is, again, mame and cores based on mame, because they handle their zip stuff internally and their zip files are never that big, so they can go into main memory (iirc).
Another exception inside a emulator core is the new dolphin support for their new compression format.
edit: and as i was reminded below, another exception is the chd format for the ps1, ps2 etc cores.

[Augusto7743]

That issue is terrible.
You has a zip file name SMB and inside with some SMB versions thus help to create a playlist easy to browse.
Stella browse the zip files without problems.
That issue need a fix.

[i30817]

Look, this isn't going to change (people have been trying to change the automatic scanner for years now but it only seems to get worse at actually scanning because people mostly complain about speed and don't give a fuck about anything not romlike, not in one of the 'big romsets' (whdloads for example), anything fan-patched or with a 'nonstandard' organization like your multiple roms in a single file).

Use the manual scanner like i said, might work that way.

Might also not work too. You won't know until you try.

Another, simple alternative is simply uncompressing all that.

This isn't actually as bad as it seems these days, unless you're playing games in a micromachine (which you might be to be fair), because those older roms were incredibly small and a single wii or ps2 game eclipses the romset of the nes or the Atari twice over.

[inactive123]

Also be aware that retroarch is simply terrible for disk wear at handling compressed content if you're not explicitly using a ramdisk for temporary files.

I feel compelled to respond to this.

I asked jdgleaver about this actually, and this is what he told me -

"Up until 5 months ago, what he says there was correct - all compressed content was extracted to a temporary file on disk before being loaded (apart from arcade and any other 'block extract' cores)
But I fixed that here: #12421
Now, provided that a core sets need_fullpath = false (either normally or via the RETRO_ENVIRONMENT_SET_CONTENT_INFO_OVERRIDE callback), the extracted content remains entirely in RAM, with no disk wear whatsoever.
Now it's only things like CD images that get extracted to temporary files (but zipping CD images is insane when you can use CHD instead....)"

So it's important to be informed that actually we have gone to great lengths to reduce wear and tear on disk I/O recently I (as of at least 5 months ago)and what is being described by you should no longer be an issue in the vast majority of cases.

[i30817]

That's good. I'd go further and add a annoying warning (or at least a command line only warning) when a user tries to load a cd image from inside a zip; but i understand that's annoying for users for as long as romsets like redump use the zip format instead of chd.

[zoltanvb]

Fixed by #18641