Setting a value to N/A

[balbinot]

I have been trying to set certain values in a dataframe to Not Available using the where() statement like in the code below

df = vaex.open(file)
tmasscols = tmass.column_names
for col in tmasscols[0:10]:
    print(col)
    df[f'{col}_new'] = df.func.where(df['tmass_oid'].isna(), np.nan,  df[f'{col}']) 

This runs, but when printing, exporting, or retrieving the values of the new columns I get an error like

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/users/balbinot/data/anaconda3/envs/vaex4a6/lib/python3.7/site-packages/vaex/execution.py", line 176, in execute_async
raise RuntimeError(f'Oops, requesting column {column} from dataset, but it does not exist')
RuntimeError: Oops, requesting column nan from dataset, but it does not exist
ERROR:MainThread:vaex:error evaluating: b_m_opt_new at rows 0-5
Traceback (most recent call last):
File "/Users/users/balbinot/data/anaconda3/envs/vaex4a6/lib/python3.7/site-packages/vaex/dataframe.py", line 3539, in table_part
values = dict(zip(column_names, df.evaluate(column_names)))
File "/Users/users/balbinot/data/anaconda3/envs/vaex4a6/lib/python3.7/site-packages/vaex/dataframe.py", line 2608, in evaluate
return self._evaluate_implementation(expression, i1=i1, i2=i2, out=out, selection=selection, filtered=filtered, array_type=array_type, parallel=parallel, chunk_size=chunk_size)
File "/Users/users/balbinot/data/anaconda3/envs/vaex4a6/lib/python3.7/site-packages/vaex/dataframe.py", line 5512, in evaluate_implementation
df.map_reduce(assign, lambda *: None, expression_to_evaluate, ignore_filter=False, selection=selection, pre_filter=use_filter, info=True, to_numpy=False)
File "/Users/users/balbinot/data/anaconda3/envs/vaex4a6/lib/python3.7/site-packages/vaex/dataframe.py", line 323, in map_reduce
return self._delay(delay, task)
File "/Users/users/balbinot/data/anaconda3/envs/vaex4a6/lib/python3.7/site-packages/vaex/dataframe.py", line 1492, in _delay
self.execute()
File "/Users/users/balbinot/data/anaconda3/envs/vaex4a6/lib/python3.7/site-packages/vaex/dataframe.py", line 306, in execute
just_run(self.execute_async())
File "/Users/users/balbinot/data/anaconda3/envs/vaex4a6/lib/python3.7/site-packages/vaex/asyncio.py", line 35, in just_run
return loop.run_until_complete(coro)
File "/Users/users/balbinot/data/anaconda3/envs/vaex4a6/lib/python3.7/site-packages/nest_asyncio.py", line 98, in run_until_complete
return f.result()
File "/Users/users/balbinot/data/anaconda3/envs/vaex4a6/lib/python3.7/asyncio/futures.py", line 181, in result
raise self._exception
File "/Users/users/balbinot/data/anaconda3/envs/vaex4a6/lib/python3.7/asyncio/tasks.py", line 249, in __step
result = coro.send(None)
File "/Users/users/balbinot/data/anaconda3/envs/vaex4a6/lib/python3.7/site-packages/vaex/dataframe.py", line 311, in execute_async
await self.executor.execute_async()
File "/Users/users/balbinot/data/anaconda3/envs/vaex4a6/lib/python3.7/site-packages/vaex/execution.py", line 176, in execute_async
raise RuntimeError(f'Oops, requesting column {column} from dataset, but it does not exist')
RuntimeError: Oops, requesting column nan from dataset, but it does not exist

If instead I try using a string instead of np.nan, this works fine.

I am using 4.0.0-alpha6

[JovanVeljanoski]

Dear @balbinot

For filling in nans, N/A or missing values please use the .fillna .fillnan or .fillmissing methods. The .where method is not suitable for this as it does not play well with missing n/a and nans to the best of my knowledge. Hence the existence of the specific methods for this kind of tasks.

I hope this helps,
Jovan.

[balbinot]

Hey,

I see, the fill*() methods don't really solve it, I am trying to do is the opposite, given a conditional expression for a column I want to set elements that satisfy that condition to N/A.

Cheers

[JovanVeljanoski]

Ah i got it.. sorry i misunderstood.
I think this was asked before as well.. not sure what the answer was if any.
Let me look into it. Thanks!