Old Cody Archive Data Posting

cocoon solr No pregenerated HTML

• Data: https://github.com/CDRH/data_cody
• Code: https://github.com/CDRH/cocoon_cody
• Dev: https://cdrhdev1.unl.edu/cocoon/codyarchive.org/
• Prod: https://codyarchive.org

This is the Cocoon implementation of the Cody Archive, which powers the current production website. In the near future it will be replaced by a Rails application.

This application has an unusual aspect, which is that if you are changing the personography, you will need to break and hard refresh the file creating JSON from a faceted response from Solr.

Update Development

Log in and pull to development

Step 1: Log in

SSH into the development server and navigate to the data repository. [More info: if this is your first time logging in, you will need to set up server access.]

ssh username@cdrhdev1.unl.edu


cd /var/local/www/data/collections/cody_archive

Step 2: Check for changes

Check to see which branch is currently checked out and see if there are any unexpected changes.

git status

Typically, you should be on the dev or main branch. [More info: What to do if it is on a different branch.]

If you're on your desired branch, but there are outstanding changes, you will need to deal with them before you pull. [More info: There are files changed on the server.]

Step 3: Pull!

Once you are on the correct branch, there are no outstanding changes, and everything is copacetic, then you can pull updates.

git pull origin [current branch name]

You should either get a message saying "Already up to date" or you should see a bunch of new files show up with no errors. [More info: What if there is a problem when I git pull?]

Step 4: Upload images and other media

If your project uses images, audio, or video, you will need to upload any files that are connected with your new documents! TODO figure out some instructions knowing that most people use SFTP clients and that this isn't always on the same server oh no.

You will need to put your files in the following location on cors1601.unl.edu:

/var/local/www/media/cody_archive/[relevant directory]

If you REPLACE any images and you do not see the new version show up on your site, this is because the IIIF image server caches images and you will need to either wait (a few days) or ask a CDRH dev to purge the cache.

Now you should be ready to begin updating the development website's contents!

Update the development site

Step 1. Clear Solr (optional)

If you have removed any files or changed identifiers, you will need to clear the old, no longer used file from the Solr index. If not, skip to Step 2.

You may either clear either one specific file or ALL the files. Keep in mind that if you clear all the files, the site will have no content until you repopulate the search.

solr_clear_index -e development

You can also clear a specific file if you only need to drop one item from the index, but know that if you use an id like `10` you may be clearing more items than you intend, so be specific if possible!

# clear one file from the index with -r [id]
#   (do not include extension)
#   (be specific with id to avoid accidentally removing more files)
solr_clear_index -e development -r wwa\.0001

[More info: other options for clearing Solr by subcategory, etc]

Step 2. Populate the search!

This is the part where you get to add your new and updated files into your site's search!

post -e development -x solr

This may take some time to run, so if you are impatient, you may want to consider only posting specific files unless if you cleared the entire index in the last step. [More info: learn about posting by file type, date, and file name.]

Check the site

This site is built in cocoon, which means that you will need to do one more thing before you can see your changes!

You are going to need to BREAK THE SITE.

This sounds scary but I promise you that it almost always, usually, well, probably is not. We need to break the site so that Cocoon will refresh its contents. We do this by introducing an intentional error in some XSLT.

cd /var/local/www/cocoon/[ your_project_dir ]

mv stylesheets/xslt/tei.p5.xsl stylesheets/xslt/tei.p5.tmp.xsl

Now, go to https://cdrhdev1.unl.edu/cocoon/codyarchive.org/ and hard refresh the page. It should be broken! That's a good thing. Now let's unbreak it.

mv stylesheets/xslt/tei.p5.tmp.xsl stylesheets/xslt/tei.p5.xsl

# move back to the data directory
cd /var/local/www/data/collections/

Go back to https://cdrhdev1.unl.edu/cocoon/codyarchive.org/ and refresh again. Hopefully it is unbroken and your changes are now available! [More info: what if it's still broken?]

If you have altered any of the personography or place entries, you will also need to break an additional file:

cd /var/local/www/cocoon/[cody path]

mv stylesheets/xslt/solr2json-facets.xsl stylesheets/xslt/solr2json-facets.tmp.xsl

Now, go to TODO BASED ON ENVIRONMENT / https://codyarchive.org/js/solr/people.json and hard refresh the page. It should be broken! That's a good thing. Now let's unbreak it.

mv stylesheets/xslt/solr2json-facets.tmp.xsl stylesheets/xslt/solr2json-facets.xsl

# mv back to the data repo
cd /var/local/www/data/collections/cody_archive

Push changes (if needed)

Part 1: Check changes

After you've run your script, check if there are any files that have changed which you should commit.

git status

In general, files generated for the development environment will not need to be committed, so do not be surprised if there isn't anything here!

Part 2: Commit and push

If there are expected changes, such as to output/development/html, or for project-specific features, you will need to commit them.

# you may run git add repeatedly to add different directories and files
git add [file path] [or directory path]

# check it over to make sure you're adding everything you need
git status

git commit -m "message about your changes"

git push origin [current branch name]

Update Production

Log in and pull to production

Step 1: Log in

SSH into the production server and navigate to the data repository. [More info: if this is your first time logging in, you will need to set up server access.]

ssh username@cors1601.unl.edu


cd /var/local/www/data/collections/cody_archive

Step 2: Check for changes

Check to see which branch is currently checked out and see if there are any unexpected changes.

git status

Typically, you should be on the dev or main branch. [More info: What to do if it is on a different branch.]

If you're on your desired branch, but there are outstanding changes, you will need to deal with them before you pull. [More info: There are files changed on the server.]

Step 3: Pull!

Once you are on the correct branch, there are no outstanding changes, and everything is copacetic, then you can pull updates.

git pull origin [current branch name]

You should either get a message saying "Already up to date" or you should see a bunch of new files show up with no errors. [More info: What if there is a problem when I git pull?]

Step 4: Upload images and other media

This section is repeated for the sake of being thorough, but if you already have added your media files either during the development step or previously, you may skip this!

If your project uses images, audio, or video, you will need to upload any files that are connected with your new documents! TODO figure out some instructions knowing that most people use SFTP clients and that this isn't always on the same server oh no.

You will need to put your files in the following location on cors1601.unl.edu:

/var/local/www/media/cody_archive/[relevant directory]

If you REPLACE any images and you do not see the new version show up on your site, this is because the IIIF image server caches images and you will need to either wait (a few days) or ask a CDRH dev to purge the cache.

Now you should be ready to begin updating the production website's contents!

Update the production site

Step 1. Clear Solr (optional)

If you have removed any files or changed identifiers, you will need to clear the old, no longer used file from the Solr index. If not, skip to Step 2.

You may either clear either one specific file or ALL the files. Keep in mind that if you clear all the files, the site will have no content until you repopulate the search.

solr_clear_index -e production

You can also clear a specific file if you only need to drop one item from the index, but know that if you use an id like `10` you may be clearing more items than you intend, so be specific if possible!

# clear one file from the index with -r [id]
#   (do not include extension)
#   (be specific with id to avoid accidentally removing more files)
solr_clear_index -e production -r wwa\.0001

[More info: other options for clearing Solr by subcategory, etc]

Step 2. Populate the search!

This is the part where you get to add your new and updated files into your site's search!

post -e production -x solr

This may take some time to run, so if you are impatient, you may want to consider only posting specific files unless if you cleared the entire index in the last step. [More info: learn about posting by file type, date, and file name.]

Check the site

This site is built in cocoon, which means that you will need to do one more thing before you can see your changes!

You are going to need to BREAK THE SITE.

This sounds scary but I promise you that it almost always, usually, well, probably is not. We need to break the site so that Cocoon will refresh its contents. We do this by introducing an intentional error in some XSLT.

cd /var/local/www/cocoon/[ your_project_dir ]

mv stylesheets/xslt/tei.p5.xsl stylesheets/xslt/tei.p5.tmp.xsl

Now, go to https://codyarchive.org and hard refresh the page. It should be broken! That's a good thing. Now let's unbreak it.

mv stylesheets/xslt/tei.p5.tmp.xsl stylesheets/xslt/tei.p5.xsl

# move back to the data directory
cd /var/local/www/data/collections/

Go back to https://codyarchive.org and refresh again. Hopefully it is unbroken and your changes are now available! [More info: what if it's still broken?]

If you have altered any of the personography or place entries, you will also need to break an additional file:

cd /var/local/www/cocoon/[cody path]

mv stylesheets/xslt/solr2json-facets.xsl stylesheets/xslt/solr2json-facets.tmp.xsl

Now, go to TODO BASED ON ENVIRONMENT / https://codyarchive.org/js/solr/people.json and hard refresh the page. It should be broken! That's a good thing. Now let's unbreak it.

mv stylesheets/xslt/solr2json-facets.tmp.xsl stylesheets/xslt/solr2json-facets.xsl

# mv back to the data repo
cd /var/local/www/data/collections/cody_archive

Push changes (if needed)

Part 1: Check changes

After you've run your script, check if there are any files that have changed which you should commit.

git status

Part 2: Commit and push

If there are expected changes, such as to output/production/html, or for project-specific features, you will need to commit them.

# you may run git add repeatedly to add different directories and files
git add [file path] [or directory path]

# check it over to make sure you're adding everything you need
git status

git commit -m "message about your changes"

git push origin [current branch name]

More Information

Login and Pull

How to set up SSH

TODO

What to do if the git branch is not dev or main

TODO

There are files changed on the server

TODO

What if there is a problem when I git pull?

TODO cover permissions, merge conflict

Update the site

Clearing Solr by subcategory and more

TODO

Clearing Elasticsearch by subcategory and more

TODO possibly combine with above

Posting to Solr by specific file type, name, and updated date

TODO

HTML generation errors

TODO

Elasticsearch population errors

TODO

Check the site

Cocoon is broken

TODO

How to hard refresh

TODO

Push changes