experimental support for SPV files now available

[Ben Pfaff]

I've been working hard to add PSPP support for reading SPV files. I've
pushed what I have to the spv-reader branch at git://benpfaff.org/pspp.

The code isn't really ready for prime time, but if you just want to see
what works, you can visit https://pspp.benpfaff.org/ to upload an .spv
file and it will instantly translate it to a PDF for viewing. (Don't do
this with your confidential data!)

I know it's not perfect, although it does a mostly adequate job with the
3000+ files in my test corpus. I'd love to have feedback.

Thanks,

Ben.

[Alan Mead]

Ben,

This is fantastic! I'm about to send you some feedback on three SPV
files. The message is about 2.7MB. I'm pretty sure I cannot send such an
email to the list.

If it doesn't arrive, let me know how I can get you send it.

Short description: Works pretty well! Which is amazing! But: Images
don't appear. Page numbers would be nice. The formatting could be
improved which is mainly an issue because of how it would print. SPSS's
export allows including non-visible nodes out output (all those "Notes"
tables). Editing the SPV caused some chaos.

-Alan

Post by Ben Pfaff
I've been working hard to add PSPP support for reading SPV files. I've
pushed what I have to the spv-reader branch at git://benpfaff.org/pspp.
The code isn't really ready for prime time, but if you just want to see
what works, you can visit https://pspp.benpfaff.org/ to upload an .spv
file and it will instantly translate it to a PDF for viewing. (Don't do
this with your confidential data!)
I know it's not perfect, although it does a mostly adequate job with the
3000+ files in my test corpus. I'd love to have feedback.
Thanks,
Ben.
_______________________________________________
Pspp-users mailing list
https://lists.gnu.org/mailman/listinfo/pspp-users

--
Alan D. Mead, Ph.D.
President, Talent Algorithms Inc.

science + technology = better workers

http://www.alanmead.org

"You're an interesting species. An interesting mix.
You're capable of such beautiful dreams, and such
horrible nightmares. You feel so lost, so cut off,
so alone, only you're not. See, in all our
searching, the only thing we've found that makes
the emptiness bearable, is each other."

-- Carl Sagan, Contact

[Ben Pfaff]

Post by Alan Mead
Ben,
This is fantastic! I'm about to send you some feedback on three SPV
files. The message is about 2.7MB. I'm pretty sure I cannot send such an
email to the list.
If it doesn't arrive, let me know how I can get you send it.
Short description: Works pretty well! Which is amazing! But: Images
don't appear. Page numbers would be nice. The formatting could be
improved which is mainly an issue because of how it would print. SPSS's
export allows including non-visible nodes out output (all those "Notes"
tables). Editing the SPV caused some chaos.

Thanks for all the feedback!

When you say "images", do you mean charts? SPV files can include .png
files and other kinds of images pasted in from elsewhere. It would not
be difficult to support them but they are rare in practice (I only found
a few in my corpus) and I haven't done the work yet. Charts, on the
other hand, are pretty common but also a lot of work and I haven't
really started on them at all yet.

I hadn't looked at page numbers yet. They are a fairly small
incremental amount of work. I'll put them on my to-do list.

I agree that the formatting is poor in places. I'm planning to work on
this too.

The pspp-output utility underlying the web service supports including
hidden objects like Notes, but I hadn't enabled it by default on the web
service. I've now added an option for this on the webpage. (Also the
webpage now allows one to request CSV or plain text output. Nifty?)

The chaos caused by editing was easily fixable, once I had an example of
the problem, and I fixed it.

[Matej Kovacic]

Hi,

I am glad to hear that PSPP is going to support SPV files soon. I do not
have SPSS, but is it possible that someone send an image of SPV file and
converted file to the list?

Maybe you should include that to your website too...

Regards,
M.

[Alan Mead]

I don't know how big a file I can post to the list. I'm attaching the an
image showing a side-by-side comparison of FREQ output.

They're not pixel identical, but the PDF format is very similar. The
difference at the top of this page in this image is because of
configuration differences (the title and dataset information appear on
the previous page, above the "Notes" section).

The files are functionally equivalent (aside from charts). I think the
font Ben used is easier to read.

-Alan

Post by Matej Kovacic
Hi,
I am glad to hear that PSPP is going to support SPV files soon. I do not
have SPSS, but is it possible that someone send an image of SPV file and
converted file to the list?
Maybe you should include that to your website too...
Regards,
M.
_______________________________________________
Pspp-users mailing list
https://lists.gnu.org/mailman/listinfo/pspp-users

--
Alan D. Mead, Ph.D.
President, Talent Algorithms Inc.

science + technology = better workers

http://www.alanmead.org

"You're an interesting species. An interesting mix.
You're capable of such beautiful dreams, and such
horrible nightmares. You feel so lost, so cut off,
so alone, only you're not. See, in all our
searching, the only thing we've found that makes
the emptiness bearable, is each other."

-- Carl Sagan, Contact

[Alan Mead]

Post by Ben Pfaff
When you say "images", do you mean charts? SPV files can include .png
files and other kinds of images pasted in from elsewhere. It would not
be difficult to support them but they are rare in practice (I only found
a few in my corpus) and I haven't done the work yet. Charts, on the
other hand, are pretty common but also a lot of work and I haven't
really started on them at all yet.

Yes, I meant charts. They're probably the biggest bit of missing
functionality (maybe there are other things I don't use)

This is really cool. I'll try the new conversion features later today.

Does this handle SPO files? I guess I can check... IIRC, there are old
file formats that current SPSS users won't be able to view at all.

-Alan
--
Alan D. Mead, Ph.D.
President, Talent Algorithms Inc.

science + technology = better workers

http://www.alanmead.org

"You're an interesting species. An interesting mix.
You're capable of such beautiful dreams, and such
horrible nightmares. You feel so lost, so cut off,
so alone, only you're not. See, in all our
searching, the only thing we've found that makes
the emptiness bearable, is each other."

-- Carl Sagan, Contact

[Ben Pfaff]

Post by Alan Mead
Does this handle SPO files? I guess I can check... IIRC, there are old
file formats that current SPSS users won't be able to view at all.

This does not yet handle SPO files. Toward adding support for them, I
have a small corpus of sample .spo files that I'll investigate at some
point. I'd appreciate donations of more .spo files.

[Alan Mead]

This does not yet handle SPO files. Toward adding support for them, I
have a small corpus of sample .spo files that I'll investigate at some
point. I'd appreciate donations of more .spo files.

I think I've sent you all mine. The demand for SPO may be small...
--
Alan D. Mead, Ph.D.
President, Talent Algorithms Inc.

science + technology = better workers

http://www.alanmead.org

"You're an interesting species. An interesting mix.
You're capable of such beautiful dreams, and such
horrible nightmares. You feel so lost, so cut off,
so alone, only you're not. See, in all our
searching, the only thing we've found that makes
the emptiness bearable, is each other."

-- Carl Sagan, Contact

[Ben Pfaff]

Post by Alan Mead

This does not yet handle SPO files. Toward adding support for them, I
have a small corpus of sample .spo files that I'll investigate at some
point. I'd appreciate donations of more .spo files.

I think I've sent you all mine. The demand for SPO may be small...

Oh, I forgot, most of my .spo corpus actually comes from you.

New .spo files do show up regularly at the converter webpage. That's
how I collected most of my 3000+ .spv files, too. (I'd have far more
.spo files if I hadn't inadvertently deleted most of them a few months
ago not recognizing the format.)

[Alan Mead]

Post by Ben Pfaff
New .spo files do show up regularly at the converter webpage. That's
how I collected most of my 3000+ .spv files, too. (I'd have far more
.spo files if I hadn't inadvertently deleted most of them a few months
ago not recognizing the format.)

I take it that SPO and SPV files are very different? How "smart" are SPV
files? I ask because I've had the experience of opening an SPV file in
SPSS and SPSS seems to be composing the results (a table or a chart). I
had the impression that it was re-running the analysis, but maybe that's
a poor assumption. But I believe I've heard reports that when people
with different licenses exchange SPV files, some results cannot be
displayed (because a particular module is not available on the viewer's
SPSS license). If so, I imagine there are limits on what PSPP can do to
display some SPO files?

-Alan
--
Alan D. Mead, Ph.D.
President, Talent Algorithms Inc.

science + technology = better workers

http://www.alanmead.org

"You're an interesting species. An interesting mix.
You're capable of such beautiful dreams, and such
horrible nightmares. You feel so lost, so cut off,
so alone, only you're not. See, in all our
searching, the only thing we've found that makes
the emptiness bearable, is each other."

-- Carl Sagan, Contact

[Ben Pfaff]

Post by Alan Mead

Post by Ben Pfaff
New .spo files do show up regularly at the converter webpage. That's
how I collected most of my 3000+ .spv files, too. (I'd have far more
.spo files if I hadn't inadvertently deleted most of them a few months
ago not recognizing the format.)

I take it that SPO and SPV files are very different?

They're very different on the outside--an SPV file is a Zip archive and
an SPO file is a MS Compound Document Format archive. I haven't looked
enough at what's inside an SPO file yet to say how different the
contents are, but I suspect that they are very different.

Post by Alan Mead
How "smart" are SPV files? I ask because I've had the experience of
opening an SPV file in SPSS and SPSS seems to be composing the results
(a table or a chart). I had the impression that it was re-running the
analysis, but maybe that's a poor assumption.

SPV files contain results, in the form of pivot tables, charts, and
other objects. Loading one does not re-run analyses, but it does load
objects from disk only as they come into view, and it does take a little
bit of CPU to display them.

Post by Alan Mead
But I believe I've heard reports that when people with different
licenses exchange SPV files, some results cannot be displayed (because
a particular module is not available on the viewer's SPSS license).

I guess that SPSS could limit what it shows however it likes, but the
pivot tables in SPV files are generic. A program that can display any
SPV pivot table should be able to display all of them.

Some SPV files contain a specialized kind of object called a "model". I
don't know anything about these yet. It is possible that displaying
models requires a special license.

Some SPV files produced by a newer version of SPSS might not display on
older versions. I've tried to implement all the features that appear in
the corpus, which includes files produced by newer and older versions of
SPSS.

Post by Alan Mead
If so, I imagine there are limits on what PSPP can do to display some
SPO files?

SPO files are a separate question. I don't know if they have the same
underlying data model as SPV files. If not, they might be hard to deal
with. I need to investigate the ones I have before I can guess. But
I'm planning to get SPV files fine-tuned before I look at them.

[Alan Mead]

Post by Ben Pfaff
SPV files contain results, in the form of pivot tables, charts, and
other objects. Loading one does not re-run analyses, but it does load
objects from disk only as they come into view, and it does take a little
bit of CPU to display them.

This explains what I was seeing, I'm sure. Charts can be the most CPU
intensive, which makes sense because something like a scatterplot
probably shows all the individual data points. Sounds like the the
problem I speculated about might not exist or only for some modules (maybe

Post by Ben Pfaff

Post by Alan Mead
If so, I imagine there are limits on what PSPP can do to display some
SPO files?

SPO files are a separate question. I don't know if they have the same
underlying data model as SPV files. If not, they might be hard to deal
with. I need to investigate the ones I have before I can guess. But
I'm planning to get SPV files fine-tuned before I look at them.

That was a typo.  IIRC SPO format superceeded text output and added
formatting; I'd be surprised if it wasn't some kind of rich text like
RTF.  If its some homebrewed format, it might be a lot of work to make
it display properly. I suspect that if it was simple to display SPO
files, there wouldn't be any issue with current versions of SPSS opening
them. But that's just a pessimistic guess.

-Alan
--
Alan D. Mead, Ph.D.
President, Talent Algorithms Inc.

science + technology = better workers

http://www.alanmead.org

"You're an interesting species. An interesting mix.
You're capable of such beautiful dreams, and such
horrible nightmares. You feel so lost, so cut off,
so alone, only you're not. See, in all our
searching, the only thing we've found that makes
the emptiness bearable, is each other."

-- Carl Sagan, Contact

[Ben Pfaff]

Post by Ben Pfaff

Post by Alan Mead
Ben,
This is fantastic! I'm about to send you some feedback on three SPV
files. The message is about 2.7MB. I'm pretty sure I cannot send such an
email to the list.
If it doesn't arrive, let me know how I can get you send it.
Short description: Works pretty well! Which is amazing! But: Images
don't appear. Page numbers would be nice. The formatting could be
improved which is mainly an issue because of how it would print. SPSS's
export allows including non-visible nodes out output (all those "Notes"
tables). Editing the SPV caused some chaos.

Thanks for all the feedback!
When you say "images", do you mean charts? SPV files can include .png
files and other kinds of images pasted in from elsewhere. It would not
be difficult to support them but they are rare in practice (I only found
a few in my corpus) and I haven't done the work yet. Charts, on the
other hand, are pretty common but also a lot of work and I haven't
really started on them at all yet.
I hadn't looked at page numbers yet. They are a fairly small
incremental amount of work. I'll put them on my to-do list.
I agree that the formatting is poor in places. I'm planning to work on
this too.
The pspp-output utility underlying the web service supports including
hidden objects like Notes, but I hadn't enabled it by default on the web
service. I've now added an option for this on the webpage. (Also the
webpage now allows one to request CSV or plain text output. Nifty?)
The chaos caused by editing was easily fixable, once I had an example of
the problem, and I fixed it.

Page numbers work now--all page headers and footers, actually. Some SPV
files don't have any configuration for page headers and footers, so they
won't show up for those, but they should work in the ones that do.