Importing Molecules from Files

[zachcran]

I am starting this discussion from PR #25, where we (@ryanmrichard, @hjjvandam, and @zachcran) discussed whether storing molecules as C++ code (as we currently do in ChemCache) is still the best approach.

This discussion continues that exchange from the last comment:

@ryanmrichard I agree with @hjjvandam that having a tool to read in XYZ files (or CML, SMILES, etc.) and using that to read the raw molecule files seems like a good path, as opposed to storing the molecules as C++ code. I agree that this would eliminate the need for MoleculeManager as well. I like your Python solution of

systems = [ nwx.system_from_file(x) for x in os.listdir(xyz_dir) ]

although that would generate C++ that need to be compiled on the fly, correct? I don't really know if that is an issue, since I do not understand the binding methods from Python to C++, but I figured I would bring it up anyway.

We would definitely need to discuss more if we go this route, including whether to make it a PyPI package. I would like to close (not merge) this PR, so maybe we can move discussion to another issue if required?

[ryanmrichard]

FWIW I don't think there's anything about the proposed Python solution which would require just-in-time compilation. The potential caveat to that is whether the system_from_file function gets written in Python or C++. If it's in C++, then the way that's written, system_from_file is going to get some sort of Python object which likely doesn't have a C++ analog (i.e., there's probably not going to be an automatic conversion to say a C++ std::filesystem::path object). That said, I'd probably just write system_from_file in Python, if for no other reason than to avoid having to do file parsing in C++.

I vote we delete MoleculeManager and implement system_from_file, in Python, and in the NWChemEx repo. The actual molecular geometries can live here as XYZ files.

[zachcran]

@ryanmrichard @hjjvandam That all makes sense, thanks! It sounds like the path forward is a system_from_file function to load molecules from files, since users will want to run different molecules often. So the todo list so far looks something like:

• Implement system_from_file in the NWChemEx repo in Python
• Delete Chemist::MoleculeManager
• Remove the corresponding generator script from this repository (and CI)

Another tangential thought, there should be a way to input a system explicitly (like how you copy/paste coordinates into an NWChem input file). Do we have something like that in the works?

[ryanmrichard]

@zachcran If it were me I'd just worry about figuring out what Python libraries are already out there for reading molecules in from files. If the user gives you a string representation of the molecule, worst case scenario, we can always write it to a file and make the molecule that way. That said, I suspect what most of the libraries actually do amounts to string parsing (you read the file into a string, and then parse the string); meaning, if the user gave us a string we could just forward the string to the parsing library.

The main reason I recommend finding a library is that IIRC there's over a 100 different chemical file formats and using a library gives us a big jump on being able to handle as many of them as possible.

[twindus]

There is Open Babel for C++. Doesn't help on the Python side, though.

[ryanmrichard]

@twindus sure it does 😉

[zachcran]

@ryanmrichard Yeah, there are a lot of chemical file formats out there. I agree that writing our own parsers for all, or even a subset, would be a nightmare, and it is already a solved problem.

Open Babel sounds like a promising option! I have had good experiences with it in the past. The source code is compiled with CMake and allows you to compile language bindings. It also looks like there is a PyPI package, so pip install openbabel might be sufficient.

[wadejong]

We will have a molecule object generator on the Python side, for example XYZ to start.

[zachcran]

@wadejong Just to clarify, does this mean that you already have a molecule object generator for XYZ files, like we are discussing above, in progress?

[hjjvandam]

Open Babel does have Python bindings as far as I know: https://openbabel.org/wiki/Python

[hjjvandam]

Weird, I don’t see the post below on GitHub (yet). Anyway, I think OpenBabel is a sensible solution. However, note that I have seen it do some really weird stuff when trying to generate a 3D structure from a SMILES string. I reported one bug and that one got fixed (OpenBabel builds the structure from fragments, and some fragments had all atomic coordinates listed as 0.0). However, more recently I got some surprising results trying to generate the structure of 2-propanol from the SMILES string CC(O)C (it generated a triangular carbon ring with a ketone on it). So in my experience, if you start from a structure that has all atoms specified then OpenBabel seems to be reliable in converting from one to another. But if you start from something where OpenBabel has to add atoms (hydrogens for example) then I would be more careful.

…

----------------------------------------------------------------------------------------------------- Hubertus van Dam, 631-344-6020, ***@***.******@***.***> Brookhaven National Laboratory, orcid.org/0000-0002-0876-3294<http://www.orcid.org/0000-0002-0876-3294> From: Zachery Crandall ***@***.***> Reply-To: NWChemEx-Project/ChemCache ***@***.***> Date: Monday, February 21, 2022 at 12:54 To: NWChemEx-Project/ChemCache ***@***.***> Cc: "Van Dam, Hubertus" ***@***.***>, Mention ***@***.***> Subject: Re: [NWChemEx-Project/ChemCache] Importing Molecules from Files (Discussion #32) @ryanmrichard<https://github.com/ryanmrichard> Yeah, there are a lot of chemical file formats out there. I agree that writing our own parsers for all, or even a subset, would be a nightmare, and it is already a solved problem. Open Babel sounds like a promising option! I have had good experiences with it in the past. The source code is compiled with CMake<https://urldefense.com/v3/__https:/openbabel.org/docs/current/Installation/install.html*requirements__;Iw!!P4SdNyxKAPE!TWnSXRzF4RHLOophG1k2IEogRRN79yaiN4z4gwzycUiijqTwSm1zmCy4KoTZ1F0$> and allows you to compile language bindings<https://urldefense.com/v3/__https:/openbabel.org/docs/current/Installation/install.html*compile-bindings__;Iw!!P4SdNyxKAPE!TWnSXRzF4RHLOophG1k2IEogRRN79yaiN4z4gwzycUiijqTwSm1zmCy4Q0ZlVdk$>. It also looks like there is a PyPI package<https://urldefense.com/v3/__https:/pypi.org/project/openbabel/__;!!P4SdNyxKAPE!TWnSXRzF4RHLOophG1k2IEogRRN79yaiN4z4gwzycUiijqTwSm1zmCy4PcGBBOg$>, so pip install openbabel might be sufficient. — Reply to this email directly, view it on GitHub<#32 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABDS7HVC36V4QSKXY6BQB43U4J343ANCNFSM5OBRV7PA>. Triage notifications on the go with GitHub Mobile for iOS<https://urldefense.com/v3/__https:/apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!P4SdNyxKAPE!TWnSXRzF4RHLOophG1k2IEogRRN79yaiN4z4gwzycUiijqTwSm1zmCy4hzk_zLs$> or Android<https://urldefense.com/v3/__https:/play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!P4SdNyxKAPE!TWnSXRzF4RHLOophG1k2IEogRRN79yaiN4z4gwzycUiijqTwSm1zmCy4XlylbjE$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[zachcran]

That is a good note to bring up... I suppose in situations like that, someone could switch to an XYZ file or other format that is more explicit about coordinates and atom locations, since Open Babel would support those as well. However, encountering one of these bugs could definitely be frustrating for a user if they didn't catch it before wasting compute time with the wrong geometry. It would be important to document the potential risk a user takes by using vague, easier-to-write formats versus an explicit format.

We could also provide an optional mechanism for the user to view the coordinates before running the calculation. A few ideas on this, in no particular order:

• Truncate the input file, so only geometry parsing is performed in a first, pre-calculation run. Essentially, I am thinking the user would comment out the rest of the input file that does the actual calculations.
• Add an optional flag when running NWX that tells it to stop after reading the geometry and output.
• Make the geometry one of the first pieces of information in the output file, so errors can be caught quickly.

[ryanmrichard]

@hjjvandam FWIW when I'm on GitHub, and looking at the first reply chain, GitHub only shows like the first three responses. After the third response it says something like "show more". Once I click that I can see all the responses. Does that solve your problem?

[wadejong]

SMILES can be a tricky thing. Pubchem does the right thing, I thought they had Open Babel in the background too.

My thoughts is that we will be using these input formats to generate a molecule object. There needs to be some responsibility on the user side to check if the resulting molecule is what they expect. Blind trust in translators is a concern, as there is no perfect solution.