asciiSetupReader

• **Column names are always unique.** When two columns would get the same clean name, the later ones get their short name added, e.g. the second "CLOTHING" column, V93, is now CLOTHING_V93, and PSID's repeated "RELEASE NUMBER" becomes RELEASE_NUMBER_TA050001 for the 2005 wave. Names are compared ignoring case, as SPSS and SAS do, so Age and AGE count as the same name. Before, the data had duplicate names, so data$CLOTHING only gave the first of them and dplyr and data.table stopped with an error. A message lists the renamed columns.
• **An = inside a label no longer cuts the name short.** "SEP FROM FATHER < OR = 5" was named X5; it is now SEP_FROM_FATHER_under_OR_5.
• **% in a label becomes _percent_** in the name, as < already became _under_ (e.g. "1919 RT % VOTES CAST" is X1919_RT_percent_VOTES_CAST, not X1919_RT_VOTES_CAST, which "1919 RT VOTES CAST" could also be).
• **Missing values are matched by value**, as SPSS and SAS do: a missing value written 9, 09 or 0000009 matches any of them in the data. Before, they had to be written exactly as in the data, so IF V156=0000009 THEN V156=.; never matched the 9s in a 1-column field. Columns the setup file says are text (SPSS (A), SAS $) still need exactly the same text.
• **Value labels of number columns are matched by value** too, so a label for 0 now labels data written 0000000 (e.g. 7,997 "Not updated" values in the 1960 UCR file that used to stay as 0000000). A column that now has some labelled values is text rather than numbers: in the 1960 UCR file, columns such as JAN_LAST_UPDATE have "Not updated" where they had 0. use_value_labels = FALSE keeps the codes.
• **Blanks and the letters NA in text columns get their value labels**: e.g. 166,650 blanks in the 1960 UCR file are now "No return received" and 15,555 in the 2000 NIBRS batch header "Inactive", where they used to be NA. (SPSS treats a blank text value as a value; SAS treats it as missing but still gives it a format's label for a blank.) A blank or NA text value without a label is still NA. In number columns a blank, NA or a lone . is missing (a lone . used to be kept).
• **Columns the setup file says are text stay text** (SPSS (A) or an A format, SAS $), as in SPSS and SAS, even when every value is a number, so codes such as ZIP code 01001 keep their leading zeros. Before, such a column became numeric with coerce_numeric = TRUE. In the SHR example data, the rarely filled-in ages of victims 5 to 11 are now text like those of victims 1 to 4, and empty text columns are character NA rather than numeric NA. Use as.numeric() on a column you want as numbers.
• **Columns without a label in VARIABLE LABELS or LABEL are kept**, named by their short name; they used to be dropped.
• **A SAS label continued on the next line is read in full**, so names that used to be cut short at the line's end are longer: e.g. the prisoners file's LFM and LFF were ..._LOCAL_AUTHORITY and ..._LOCAL_AUTHORITY_LFF (one name, made unique) and are now ..._LOCAL_AUTHORITY_MALE and ..._LOCAL_AUTHORITY_FEMALE. 24 names in 6 of the 157 setup files used to check this release changed this way.
• **A later VALUE LABELS command replaces a column's earlier labels**, as in SPSS (ADD VALUE LABELS adds to them instead). Before, the first set was kept. A message lists the columns whose earlier, different labels were replaced.
• **SPSS GET DATA files are read at the right positions.** Their column positions count from 0, so every column used to be read one position too early. As in SPSS, GET DATA formats such as F7.1 don't imply decimal places.
• Setup files that aren't valid UTF-8 are read as Windows-1252, so curly quotes and dashes in labels are read correctly (see below).
• asciiSetupReader now needs R 4.1.0 or later (its dependencies vroom, readr and stringr already did).

Upgrading from 2.6.0

• Code that uses descriptive column names may need the new names (made unique, %, =, full SAS labels). parse_setup(setup_file)$setup lists every column's short name (column_number) and descriptive name (column_name); short names haven't changed, and use_clean_names = FALSE gives them.
• Columns with implied decimal places now have their decimal point. To compare with results from 2.6.0, multiply such a column by 10^decimals, using its decimals in parse_setup(setup_file)$setup.
• Columns can now be text rather than numbers because more of their codes get value labels, or because the setup file says they're text. use_value_labels = FALSE keeps the codes, and as.numeric() turns a text column of numbers into numbers.
• Missing values are applied as before by default (missing_values = "all", including ones that are commented out in the setup file).

New features

• read_ascii_setup(), spss_ascii_reader() and sas_ascii_reader() have a missing_values argument. Many setup files have their missing values section commented out (* MISSING VALUES in SPSS, inside /* */ in SAS) so the codes stay in the data unless you choose otherwise: "all" (the default, as before) uses every missing value in the file, "active" only those that aren't commented out, and "none" none of them. When commented-out missing values are used, a message says where they are and how many values they made NA. parse_setup()$missing has a new commented column.
• A message tells you when a column with implied decimal places has values that equal one of its value labels' or missing values' codes only before the decimal point is added (e.g. -9 in a (2) column is -0.09, not the code -9 'Blank'). As in SPSS and SAS the code isn't used, but it suggests the data doesn't match the setup file's decimal places.
• Messages tell you about implied decimal places, commented-out missing values, renamed columns, replaced value labels and setup files that aren't UTF-8. They have the class "asciiSetupReader_message" (and one for each kind), so they can be silenced with suppressMessages(). Missing values that also have a value label keep the label (as before); the documentation now says so.
• More SPSS and SAS syntax is understood: VALUE LABELS and MISSING VALUES for lists of variables (V1 V2 1 'Yes', V1 TO V5 (9)) and starting on the command's own line, ADD VALUE LABELS, several MISSING VALUES sections in one file, commented-out VALUE LABELS between other sections, variables named VALUE or LABELS, and SAS formatted input (@1 V1 $CHAR2. @3 V2 4.1).
• Setup files that aren't valid UTF-8 are read as Windows-1252 (Latin-1 plus curly quotes and dashes), with a message, instead of failing with "input string 3 is invalid" (GitHub #5).
• SPSS MISSING VALUES follow SPSS's rules more closely: a later command replaces a column's missing values, V1 () clears them, ALL means every column, / can separate columns on one line, THRU can be in any case, and long or decimal ranges (1000000 THRU 9999999, 99.5 THRU 99.9) are compared as numbers instead of being listed one value at a time.
• SAS missing value recodes are read as whole statements, from every section after INPUT: >=/GE alone (98 thru highest), <=/LE, GE a AND LE b, EQ, lower-case names and statements over several lines. Statements that aren't "IF v = ... THEN v = .;" (e.g. a recode to another value, or a condition on another column) used to be misread as missing values; they're now skipped, with a message.
• Statements that change values rather than mark them missing (SAS IF v = 1 THEN v = 5;, SPSS RECODE v (1=5)) aren't applied; a message lists them. Only a setup file's layout, names, labels and missing values are used, as before, but such statements used to be misread.
• SKIP (SPSS DATA LIST), /FIRSTCASE (GET DATA) and FIRSTOBS (SAS INFILE) skip header lines in the data.
• SAS formatted input without @ pointers (INPUT V1 2. V2 $3.;), INPUT on the same line as the first column, blank lines inside INPUT and commented-out INPUT statements are handled.
• Setup files the package can't read now say why, with an error of class "asciiSetupReader_error": a file for delimited data (GET DATA /ARRANGEMENT = DELIMITED, DATA LIST FREE or LIST, or SAS DLM=/DSD, GitHub #13), a file of value labels without column positions (such as PSID's separate formats files), a file without a DATA LIST or INPUT section, or a layout the package can't read: several records per case (RECORDS=, /2, SAS #2), columns given by format only (ID (F3) NAME (A5)) or SAS variable lists in parentheses. These used to fail with "invalid 'row.names' length" or "argument of length 0", or read the wrong columns.
• SAS FORMAT statements are read as whole statements: one format for a list of columns (FORMAT V1 V2 YNF.;), numbered and positional lists (V1-V5, Q08-Q10, A1_1-A1_3, V1--V5), formats without value labels (8.2, DATE9.) and ATTRIB ... FORMAT= (also after LABEL='...'). Format names match in any case, and a format defined twice uses the later definition, as in SAS.
• Column labels are read by their quoting, so they can be laid out any way SPSS and SAS allow: in SPSS VARIABLE LABELS (or VAR LABELS), several columns on one line, / between columns, the first column on the command's own line and labels joined with +; in SAS LABEL, several columns on one line, blank lines, labels over several lines, a ; inside a label, LABEL after another statement on its line, and ATTRIB V1 V2 LABEL = '...' (for every column listed) whatever else ATTRIB sets. A LABEL statement inside a /* */ comment isn't used.
• SAS code lists (VALUE F 1, 2, 3 = 'Low';) give each code the label. Value ranges (1 - 97 = 'Actual hours') and OTHER aren't applied, so the real numbers are kept; a message lists them.
• SPSS DATA LIST is read token by token, so its spacing and layout don't matter: tightly spaced columns (AA 1-4(A) BB 5-8(A)), formats and decimals anywhere on a line (V1 1-5 (F,2) V2 6-7), columns on the DATA LIST line itself, a name and its position on different lines, /* */ comments, names with @, #, $ or ., and a missing closing period. As in SPSS, names listed before one range share it equally (A B C 1-6, Q1 TO Q3 3-5). GET DATA formats such as AUTO and ADATE10 and /VARIABLES= without spaces are read, and a second record given by a bare / stops with the several-records error. All 157 setup files used to check this release are read exactly as before.

Package

• The setup and data files used only by the tests moved from inst/testdata to tests/testthat/testdata, so they're no longer installed with the package (about 4 MB less).

Bug fixes

• select_columns: a column number past the last column stops with an error instead of crashing R (negative numbers still drop columns); a column asked for twice is read once, with its implied decimals; a name that isn't found is listed, with a note on which names changed in this version; and selecting a name that another column also had says what that column is called now.
• A missing value range that no value in the data falls in (SPSS 97 THRU HI, LO THRU -1, SAS IF V1 >= 97 THEN V1 = .;) no longer stops read_ascii_setup() with "'names' attribute [1] must be the same length as the vector [0]".
• SAS column input (INPUT V1 1-2 V2 $ 3-4 V3 5-8 .2;) is read token by token, as formatted input is, so a statement or comment after the ; on INPUT's line (INPUT V1 1-2 V2 3-4; RUN;), a /* */ comment inside INPUT and single-spaced columns (SEX $ 8 INCOME 9-15) no longer drop or misplace columns.
• Every missing value range is compared with the data as numbers, so values between whole numbers are in it too: 97.5 is missing for 97 THRU 99. parse_setup()$missing still lists a range of whole numbers value by value.
• Value labels keep apostrophes. Labels are now read by their quoting, so a quote character inside a label is part of the label: before, an apostrophe became a space or was dropped ("anti-jehovah's witness" became "anti-jehovah s witness" and 'Don''t know' became "Dont know"). This affected both SPSS and SAS setup files.
• Value labels are kept exactly as written between their quotes in other ways too: a double quote inside a single-quoted label ('Wife/"Wife"') is kept, and so are = and ;, which used to become a space or be dropped.
• A variable whose value labels are written more than once in a setup file no longer gets its name added to a label (e.g. "msa state police V9"). Within one VALUE LABELS command, later copies only add codes that weren't already labelled; a later command replaces the labels (see above).
• Fixed labels being assigned to the wrong codes in some setup files: a label of just 'Wife' (2 'Wife' 3 'Common-law husband' gave a code "2 Wife 3"), a code written '''', quoted words inside a label ('farmer, ''profits''', 'Twelfth grade, "High School", GED'), and SAS labels split over several quoted pieces, whose end became a separate code.
• SPSS value labels continued with + (at the end of a line or the start of the next) are joined into one label, and a double-quoted code followed by more pairs on the same line ("02" "two" 3 'three') no longer runs into the next pair.
• The 2017 YRBS SADC SAS setup file, which failed with "NA/NaN argument", can be read.
• A label of only punctuation (e.g. "...") no longer ends the list of column names early; that column keeps its short name.
• A setup file that only mentions MISSING VALUES in a comment no longer fails.
• SAS setup files with many value labels (such as PSID's) are parsed faster.