It is currently 20 Aug 2019, 15:49




 Page 19 of 23 [ 452 posts ]  Go to page Previous  1 ... 16, 17, 18, 19, 20, 21, 22, 23  Next
Author Message
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 11 Apr 2018, 01:16 
Honest fan
Honest fan
User avatar

Joined: 12 Sep 2015, 05:57
Posts: 68
Location: United States
Has thanked: 1 time
Been thanked: 1 time
For anyone joining late game as I have, the ld-decode programs with the suffix "32" are for the 32mhz captures using the domesday duplicator setup, and *not* for 32-bit operating systems, which was my first interpretation.
Quote:
Quote:
You might want to have a look at the PAL transform decoder, which should give superior performance on that system.

Thanks, I'll try and digest that info. The math for most of this stuff makes my brain melt though. If anyone knows of a "Quadrature Decoding and Comb Filtering for Dummies" reference, it'd be appreciated.


from Jim's own page (http://www.jim-easterbrook.me.uk/):
"I invented the world's best PAL decoder"

I am wondering if the work (software PAL comb filter) isn't already done, by Jim Easterbrook himself: https://github.com/jim-easterbrook/pyctools-pal

(megapixie posted a similar link to that code in the google doc associated with ld-decode)

In the current form, it seems to work with straight video files. I have been trying to get all the pre-reqs installed to test out the script, but am running out of time today. Not sure if I have sufficient python "chops" to make this work directly with the output of ld-decode, but that would prevent the extra step of having to encode a video file first.
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 21 Apr 2018, 20:20 
Honest fan
Honest fan
User avatar

Joined: 11 Sep 2017, 14:24
Posts: 85
Location: Sweden
Has thanked: 1 time
Been thanked: 34 times
I've just pushed a pre-release of the new Domesday Duplicator software to github:

https://github.com/simoninns/DomesdayDuplicator

This version has many improvements and bug-fixes and includes new GUI application code, Cypress FX3 code and FPGA code (you will need to update all three). The release is compatible with the 2_0 revision PCB (you must set DC compensation to on though in the GUI) and the github has the new revision 2_2 PCB and Kicad schematics. The revision 2_2 board is awaiting test though (I'm waiting for PCBs to be delivered from China before this can begin).

The new software supports 8xFSC sampling for NTSC (at 28.8 MSPS) and PAL (at 35.5 MSPS) rather than the fixed 32MSPS sampling of the previous version. There's also a lot of improvement around error handling to make the GUI usable without needing the debug output when things go wrong with device communication.

I've also revamped the build environment (and ironed out some multi-threading issues) so the GUI can now be built in release mode from the Ubuntu command line (makes it simple to build and gives it an Ubuntu look-and-feel):

Attachment:
DdD_GUI_application.jpg
DdD_GUI_application.jpg [ 134.56 KiB | Viewed 1341 times ]
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 21 Apr 2018, 20:25 
Honest fan
Honest fan
User avatar

Joined: 11 Sep 2017, 14:24
Posts: 85
Location: Sweden
Has thanked: 1 time
Been thanked: 34 times
Here's a sneak-peak of the 2_2 PCB. It's smaller (to get the length under 10cm which reduces the cost for fabrication since 10cm is usually the 'price-break' point) and has been much more carefully laid out to make it easier to construct. The silkscreen is also greatly improved to make it both better looking as well as clearer as to where the components go. It also has a fix for the DC offset issue, but that is subject to test (however the board can be configured to act just like the 2_0 version if needed). All the documentation has also been updated on https://www.domesday86.com

Attachment:
Domesday Duplicator V2_2 small.jpg
Domesday Duplicator V2_2 small.jpg [ 139.88 KiB | Viewed 1338 times ]
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 23 Apr 2018, 15:35 
Hardcore fan
Hardcore fan
User avatar

Joined: 18 Apr 2012, 18:02
Posts: 1506
Location: United States
Has thanked: 35 times
Been thanked: 28 times
Very cool - hopefully I'll find time to load up the new software soon! (I've been pretty busy lately.)

I don't have the mental/personal bandwidth to do this, but I just read a hackaday article on the Fresco Logic USB3 VGA adapters - they have SDR-level control over the datastream and can do >100mhz over USB3, so in theory it would be possible to put Laserdisc RF out into the player using that.

More practically you could make a perfectly timed 4/8fsc composite or s-video output with it, perhaps using dithering or blending to simulate 9-10 bit depth.
_________________
Happycube Labs: Where the past is being re-made, today. [meep!]
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 25 Apr 2018, 14:07 
Third post and above
Third post and above
User avatar

Joined: 02 Mar 2018, 11:12
Posts: 4
Location: New Zealand
Has thanked: 0 time
Been thanked: 0 time
happycube wrote:
More practically you could make a perfectly timed 4/8fsc composite or s-video output with it, perhaps using dithering or blending to simulate 9-10 bit depth.


So you can play it on the TV? Now that is a very interesting idea! :o

BTW, I don't know if anyone has mentioned it, but I remember LaserMan mentioning how he would have liked to use bandwidth expansion on the X0 project way back in time.
It sounded like a great idea for lowband LD video and I know the latest Analog Devices comb filter decoder ADV7842/ADV7850 uses it for analogue sources. No idea how you would implement that in s/w though. I know that Faroudja used the YCP101 IC in their (extremely old) VS50 video processor which seems to do a nice job on sharpening up the chroma transitions for LD.


This has to be the most interesting LD related project that I've seen in a very looooooong time.
Keep up the excellent work dudes!
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 01 May 2018, 05:43 
Third post and above
Third post and above
User avatar

Joined: 01 May 2018, 05:39
Posts: 4
Location: United States
Has thanked: 0 time
Been thanked: 0 time
I've been following this topic on and off for some time over the last year, and I have to say it's really fascinating :)

I know this topic is primarily about making a copy of the (analog) video, but in case anyone is an expert at creating a perfect dump of the digital portion of LaserActive discs, we would love to preserve this info in the redump.org dat http://forum.redump.org/post/60354/#p60354 We're a disc-based preservation group, with hashes for over 50,000 perfect dumped discs in our database.

Thank you guys for your hard work, and I look forward to seeing you come to a solution for capturing the raw video signal. Hopefully, coupled with the digital data, LaserActive games can one day emulated by MAME, etc
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 01 May 2018, 05:59 
Honest fan
Honest fan
User avatar

Joined: 11 Sep 2017, 14:24
Posts: 85
Location: Sweden
Has thanked: 1 time
Been thanked: 34 times
Quote:
I know this topic is primarily about making a copy of the (analog) video


That's not entirely the case :) The digital information on the disc is also encoded in the analogue modulated signal using a technique called EFM. If you capture the raw RF you get everything on the disc without exception.

This is the reason why I designed the Domesday Duplicator for my project to preserve the BBC Domesday system:

https://www.domesday86.com/?page_id=978

Although Domesday is PAL; the duplicator works with both NTSC and PAL. However... capturing the raw RF preserves the information on the disc, but it doesn't allow you to simply read it; for that you need a decoder (which is what ld-decode is about) - right now there is no capability to decode the EFM data, but it could be added. The details of how this is done is contained in the following (commercial) IEC specifications:

IEC 60856-1986 Laservision PAL
IEC 60856-1986 Laservision PAL Amendment 1
IEC 60856-1986 Laservision PAL Amendment 2
IEC 60857-1986 Laservision NTSC
IEC 60857-1986 Laservision NTSC Amendment 1
IEC 60857-1986 Laservision NTSC Amendment 2
IEC 60908-1999 CD digital audio system

The last spec about CDs is because the EFM encoding was developed for CDs, but used for LaserDiscs to provide both digital data and digital audio.

I've written a high-level description of the way information is decoded by a LaserDisc player which you may find useful:

https://www.domesday86.com/?page_id=1379

/Simon
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 09 May 2018, 22:49 
Third post and above
Third post and above
User avatar

Joined: 01 May 2018, 05:39
Posts: 4
Location: United States
Has thanked: 0 time
Been thanked: 0 time
simoni wrote:
The digital information on the disc is also encoded in the analogue modulated signal using a technique called EFM. If you capture the raw RF you get everything on the disc without exception.

This is the reason why I designed the Domesday Duplicator for my project to preserve the BBC Domesday system


Fascinating. So does that mean that LaserDiscs (particularly LaserActive discs) could be dumped perfectly, so that all information is preserved? And the result is repeatable (same checksums)?

simoni wrote:
capturing the raw RF preserves the information on the disc, but it doesn't allow you to simply read it; for that you need a decoder (which is what ld-decode is about) - right now there is no capability to decode the EFM data, but it could be added.


Very interesting, in preservation, step 1 is all about getting a perfect backup copy, with repeatable results (checksums), thereafter the roms will be safe for future emulation.
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 10 May 2018, 04:12 
Honest fan
Honest fan
User avatar

Joined: 11 Sep 2017, 14:24
Posts: 85
Location: Sweden
Has thanked: 1 time
Been thanked: 34 times
Quote:
Fascinating. So does that mean that LaserDiscs (particularly LaserActive discs) could be dumped perfectly, so that all information is preserved? And the result is repeatable (same checksums)?


Yes and no. You are not considering the analogue nature of a LaserDisc - Yes all the information stored on a LaserDisc can be copied 'perfectly' with all information preserved - but the copy is what is produced by the LaserDisc player's laser.

A LaserDisc is a big ol' spinning, jittering, scratched, dusty medium - so every read will naturally have variance. The disc will never spin at quite the same RPM and the laser will never track quite the same micrometre of disc track (there are a few kilometres of track on a disc - something like 70km iirc). If the disc is old, there is also the possibility that it will jump/skip during playback.

The medium is analogue - so 'perfect' is not something that will ever apply to it :)

After timebase correction the results should be more uniform though - but that's a post-process.

A lot of this reasoning is why I added 'player integrated capture' into the Domesday Duplicator software - this gives the possibility to analyse the capture on the fly and make the player go back and repeat captures for parts of the disc where there is a lot of noise or skipping occurred. There is also the possibility of combining multiple-disc captures into a single 'perfect' copy. However all of this is depending on host-side software that doesn't exist yet :)
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 10 May 2018, 05:56 
Third post and above
Third post and above
User avatar

Joined: 01 May 2018, 05:39
Posts: 4
Location: United States
Has thanked: 0 time
Been thanked: 0 time
I was under the impression there is a definite digital portion of the discs and a definite analog portion (video), based on posts like this one: http://gendev.spritesmind.net/forum/vie ... 7&start=45

I understand analog data cannot be perfectly dumped, but if the digital data can be, it would be nice to have that dumped and verified. I'm hoping you can comment strictly on the "digital" part of the LaserDisc format (the analog portion of LaserActive is a separate entity to be tackled on it's own if I understand correctly). Particularly, I'm asking on behalf of redump.org video game disc preservation group. We have datted over 50,000 unique game discs for perfect dumps. So I'm curious if the "digital" portion of the data can be perfectly dumped/redumped/verified with the same repeatable matching checksums.

Sorry if I'm sounding like a broken record, and thank you for your advisement.
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 10 May 2018, 08:08 
Honest fan
Honest fan
User avatar

Joined: 11 Sep 2017, 14:24
Posts: 85
Location: Sweden
Has thanked: 1 time
Been thanked: 34 times
It's always more complex than you think :)

Yes; you can extract the data from the sample (and the data, if error free, will always be 100% the same) - so that is true.

However, there is only one 'track' on a laserdisc and it's analogue. Every piece of data/video/sound/etc has to be in this one track. The signals for all the parts are combined using a technique known as modulation before the disc is recorded; through this all the various elements become one signal and this is what the laser reads (and therefore what the resulting RF sample contains).

There is a 'preservation distinction' here between two elements: 1) what does the RF sample contain and 2) how can I decode it into what I want?

The only way to preserve a laserdisc is to make a RF image (which is effectively an optical image of the disc's surface) at a higher resolution than used when the disc was recorded - this will vary from capture to capture and never be 'perfect' because it's analogue.

The next step (which has absolutely *nothing* to do with preservation imho (although it is useful for other things)) is decoding the RF sample back into parts you can use (data for example, for use in an emulator).

With analogue mediums it's important to make this distinction as, if there is something on the disc that you don't know about, you'll fail to 'preserve' it when you make a decoded copy (a great example is VBI frame data which isn't included in the EFM data - the computer may rely on frame numbers (VBI) to make the application work (EFM)... but it's not in the same 'place' even though its data too).

I realise this may sound like preaching but if I had a dollar for every time I was asked "why not just RGB capture the video?" or "why not just dump the data from computer?" - I'd be very rich :) The act of preservation is the reason why I designed the Domesday Duplicator; decoding it into something useful is less interesting for me; although I'm sure there are many emulator fans who will jump at the chance once RF captures become available (and I applaud and encourage such efforts!).
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 10 May 2018, 08:14 
Shows curiousity
Shows curiousity
User avatar

Joined: 20 Jan 2010, 23:33
Posts: 21
Location: United Kingdom
Has thanked: 14 times
Been thanked: 0 time
From my understanding there are a few stages that need to be complete to be able to get a digital stream out of the disc that can be verified with checksums, there is more to it than there looks at initial glance.

The signal needs to be sampled at high enough resolution - Domesday Duplicator (Thanks Simon)

The sample needs to be demodulated to recover the various streams (Video - audio - data) LD-Decode (Thanks Happycube)

Video needs to be decoded to recover VBI information (Vertical Blanking)
which includes disc information + frame / timecode which is used to detect if the player has skipped / jumped during the sampling
(Simon had some VBI decoding working but not complete)

Once it is known that the video frames are contiguous you can then use then demodulate the EFM data

Decoding the EFM to output the digital data (Not Yet Done)

Checksum the resulting data.

The only issue is that if the player has missed a few video frames then then EFM will not be contiguous and you would need a second sample of the disc where the frames have been read and update the software to extract smaller sections of data.Not as simple as it sounds as the raw sample is many GB.

The trick would be to make a file of metadata for each sample that stores the raw sample position for each of the video frames along with information about the lead-in. This would then allow you to have two copies of the disc sampled and fast seek to the correct raw sample location in any of the multiple copies to decode the data required.

With this technique and if you had three copies it may even be possible to do error recovery on the fly but that is probably many months away.
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 10 May 2018, 08:14 
Shows curiousity
Shows curiousity
User avatar

Joined: 20 Jan 2010, 23:33
Posts: 21
Location: United Kingdom
Has thanked: 14 times
Been thanked: 0 time
Sorry Simon, Both must have been writing at the same time! :-p
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 10 May 2018, 16:08 
Third post and above
Third post and above
User avatar

Joined: 01 May 2018, 05:39
Posts: 4
Location: United States
Has thanked: 0 time
Been thanked: 0 time
Thank you both for your replies!

smally_uk wrote:
From my understanding there are a few stages that need to be complete to be able to get a digital stream out of the disc that can be verified with checksums, there is more to it than there looks at initial glance.


The idea that a repeatable checksum may be possible for at least a portion of the disc is superb. I hope when this comes to pass you guys will reach out to us http://forum.redump.org/post/60354/#p60354 we would love to dat what we can perfectly dump and some of us have LaserActive discs as well.

>I had a dollar for every time I was asked "why not just RGB capture the video?" or "why not just dump the data from computer?" - I'd be very rich

I assure you that we at redump understand complexities of many discs. LaserDisc seems to be a very separate specialty altogether so your work is greatly appreciated.

I'd love to see a full LaserActive set on archive.org once your software is complete. I'm sure with a fullset of LaserActive games "best possibly dumped" would like a fire under some emulation enthusiast out there to get started on an app, if not MAME.
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 11 May 2018, 15:29 
Hardcore fan
Hardcore fan
User avatar

Joined: 18 Apr 2012, 18:02
Posts: 1506
Location: United States
Has thanked: 35 times
Been thanked: 28 times
That'd be awesome... I played a tiny bit with EFM but never got it really worked out. Especially on LD the filtering's quite difficult to work with, but maybe the Domesday Duplicator samples have better 0-2mhz quality. I'll capture some CD EFM samples and share them (this weekend? some unknown future? i dunno!)

(Never forget that EFM/CD capturing with checksum verification would be a holy grail for CD ripping...)

One related idea I've had for bad rotters (i.e. Let It Be) is that you could get a couple of copies of the same mastering and merge them to reduce/eliminate rot and boost PQ. This is much harder than it sounds (and given some of the disks that could use the treatment are rare, that ain't easy to begin with) because every capture has different timing as mentioned above, so you need to track what samples go where on both caps.

I upgraded my DD firmwares and software this week (finally!) and have some GGV1069 captures to look at this weekend. My CLD-V2800's power switch seems to be in desperate need of de-oxit-izing (and/or the power board needs a replacement and/or recap)

(BTW, if you have Anaconda installed, it has a rogue version of qmake which drops it's own QT binaries in, so it needs to be removed from the PATH on Linux before building the software. That explains the qtserial issue I had earlier... :) )
_________________
Happycube Labs: Where the past is being re-made, today. [meep!]
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 14 May 2018, 02:23 
Shows curiousity
Shows curiousity
User avatar

Joined: 18 Jul 2016, 05:18
Posts: 28
Location: Australia
Has thanked: 0 time
Been thanked: 7 times
I've had this on ice for a few months, but I'm getting back into it today. My Domesday Duplicator is fully assembled, and I'll be working on getting it programmed and tested this afternoon, so I'll see what I can get in terms of captures. I'm also planning to get back to colour decoding once I'm able to make my own rips, and then onto digital audio decoding.

For anyone else who's taking the RF signal from the test header, I'd suggest giving this cable a go:
https://au.rs-online.com/web/p/products/2967747
Making a good cable which connects coax straight to header pins isn't trivial, and this is a professionally made cable that does exactly that at a good price. This is what I've picked up and what I'll be using. If you want to add a port onto the case, you can still use this cable, just get a female to female socket that allows you to plug the BNC connector into the socket on the inside of the case, while exposing a plug on the outside, like this:
https://au.rs-online.com/web/p/products/1438920
That'll give you a professional connector without any soldering or crimping, for around $20USD. Hard to argue with that.
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 14 May 2018, 03:26 
Shows curiousity
Shows curiousity
User avatar

Joined: 18 Jul 2016, 05:18
Posts: 28
Location: Australia
Has thanked: 0 time
Been thanked: 7 times
So because having to "compile the world" under Linux is obscure and painful, I've done a build of the latest version of the software to run under a Windows environment. Only minor project changes are required to make this work. Assuming you have already assembled and programmed your Domesday Duplicator (Windows instructions for that possibly coming soon), here's how you get it running on Windows:

1. Download "zadig-2.3.exe" from here: https://zadig.akeo.ie/downloads/zadig-2.3.exe
2. Connect the USB3 connector from the Domesday Duplicator to a USB3 port on your Windows PC
3. Run zadig-2.3.exe. You should see something like the following:
Image

4. Click the "Install Driver" button. You'll only have to do this once, not each time you connect the device. If all goes well, you should see this:
Image

5. Download and extract Domesday Duplicator for Windows here: http://nemesis.exodusemulator.com/MegaLD/DomesdayDuplicatorWin.7z
6. Run DomesdayDuplicator.exe. Up should come the Domesday Duplicator interface, like so:
Image

At some point I'll do a pull request to merge back some changes so the project can cross-compile on Windows or Ubuntu without changes.
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 14 May 2018, 12:01 
Honest fan
Honest fan
User avatar

Joined: 11 Sep 2017, 14:24
Posts: 85
Location: Sweden
Has thanked: 1 time
Been thanked: 34 times
Nice work on the Windows port! Looks promising :)

Since there is a lot of discussion and interest around the Domesday Duplicator (as well as the Domesday86 project as a whole) - I've just made a facebook group for the project where people are welcome to ask questions and seek support. Anyone interested is welcome to join: https://www.facebook.com/groups/2070493199906024/

Somehow I need to control all the emails and PMs I've been getting recently :)
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 14 May 2018, 12:30 
Shows curiousity
Shows curiousity
User avatar

Joined: 18 Jul 2016, 05:18
Posts: 28
Location: Australia
Has thanked: 0 time
Been thanked: 7 times
It's Alive!!!
Image
That's a clip from "J.B. Harold - Blue Chicago Blues" for the Laseractive, decoded from RF using ld-decode, with framing and TBC carried out by my composite video decoder. Unfortunately the framing/TBC code in ld-decode fails with this kind of interleaved video stream. My code was able to lock onto it, but I don't have colour decoding yet, which just gives me more reason to focus on getting that working.

Simoni, with the 2.2 PCB revision for the Domesday Duplicator hardware, would that give a cleaner signal? I've got a stack of 30 or so disks, and I'm trying to decide if I should go ahead and rip them all now, or crank out a 2.2 PCB first and use that instead. I understand you're still waiting to test that board so I won't make one up right away, but do you expect the revised board will give a better result than a 2.0 PCB with DC offset compensation turned on?
Offline
 Profile  
 
 Post subject: Re: (WIP) Laserdisc software image decoder from raw signal
PostPosted: 14 May 2018, 12:42 
Honest fan
Honest fan
User avatar

Joined: 11 Sep 2017, 14:24
Posts: 85
Location: Sweden
Has thanked: 1 time
Been thanked: 34 times
The new board is now tested and in release status. Details (and the testing results) are available here:

https://www.domesday86.com/?page_id=1836

The new board is almost functionality identical to the previous with the exception of better DC offset accuracy (the new design is +-30mV compared to +-130mV on the previous version of the board) - this could be improved by 'matching' the resistors by hand, but 30mV in a range of 2000mV is perfectly acceptable for the analogue front-end (IMHO).

Basically this gives you a bit more 'headroom' in the RF sampling - but it probably isn't worth building a new board for (which is why I put the option to turn on the DC offset in the GUI, to keep compatibility).

Most of the new board improvements were to get the size down below the 10cm limit and make it easier to build (better component layout and a better silkscreen). The major improvements in the release were all in the software.
Offline
 Profile  
 
Display posts from previous:  Sort by  
 Page 19 of 23 [ 452 posts ]  Go to page Previous  1 ... 16, 17, 18, 19, 20, 21, 22, 23  Next


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to: