ENH Add check for non binary variables in OneHotEncoder.

[cmarmo]

Reference Issues/PRs

Fixes #16552
Closes #16554 (as superseded)

What does this implement/fix? Explain your changes.

Adds a check on the value of drop_idx_ elements: None is the value for no dropping.

[amueller]

Thanks for the PR! I haven't entirely followed the original issue so I might be missing something. Generally it would be good to have a regression test, i.e. a test that shows that now the generated feature names are as expected with your fix.

Cheers,
A.

[glemaitre]

@amueller @jnothman
As I discussed in the issue, I think that it would be best to have None as a sentinel when not having any index. Basically, we could have:

np.array([None, None, 10, 1], dtype=object)

instead of

np.array([-1, -1, 10, 10], dtype=np.int_)

The reason being that -1 could be thought to be negative indexing in Python.

So if we should agree with any solution and then we can go ahead.

ping @ogrisel @jeremiedbb @NicolasHug @thomasjpfan as well.

[cmarmo]

The reason being that -1 could be thought to be negative indexing in Python.

So if we should agree with any solution and then we can go ahead.

Indeed, see also #16593 (that I would rather label as a New Feature, but maybe I'm missing something...). If we want to use indexes in get_feature_names then the None solution should be preferred. Could someone give me a green light on that, please? Thanks!

[cmarmo]

As I would like to move forward in #15706, I have implemented drop_idx_ as array of objects, as long as I cannot use masked array I honestly prefer this solution. Some core-dev available for review? Thanks!

[jeremiedbb]

I'm ok with this change but since it will change the result of a public attribute, shouldn't it pass through a deprecation cycle ? unless we consider it a bug :)

[thomasjpfan]

The convention of setting -1 in drop_idx_ has not been released yet so we can still change it. I am okay with using None.

[NicolasHug]

@rth @thomasjpfan this was discussed during #16245 right? Could you please remind us the pros and cons of each approach?

[rth]

this was discussed during #16245 right? Could you please remind us the pros and cons of each approach?

The sentinel value itself shouldn't matter, I think, other than for readability. -1 allows to keep the dtype as int. I also have a slight preference for None as was done at some point in that PR.

[cmarmo]

Hi @rth, hope you don't mind I've just tried the "Request for reviewers" button ... :)
Thanks for you patience!

[cmarmo]

Thanks @thomasjpfan !

[jnothman]

In general I think arrays of mixed type are a strange beast. They're not especially helpful as arrays, and should rather be a list. I don't really see the problem in using -1 as a sentinel, as long as it's well tested. But the consensus seems to be towards another option.

[thomasjpfan]

In general I think arrays of mixed type are a strange beast. They're not
especially helpful as arrays, and should rather be a list. I don't really
see the problem in using -1 as a sentinel, as long as it's well tested. But
the consensus seems to be towards another option.

I can't really judge how many users will think -1 means negative indexing. I would want to go with the route that is less confusing for users. (If only we can do a poll)

[cmarmo]

@jnothman @thomasjpfan , I will not try to persuade you , just want to add a clarification.
The reason I'd rather the None solution is that -1 and negative indexes have a specific meaning in python and I don't like the idea of "overloading" it in the OneHotEncoder: one day this -1 will maybe be useful (eg a last drop option ... ok, this is a stupid example, but who knows?). Now I stop bothering you. Thanks for listening.

[jnothman]

Shrug. I'm fine either way. I agree usability is likely the greater concern here than efficiency or succinct representation. In numerical computing it is not at all uncommon when working with positive ints to give negative numbers special meanings. For instance, see how leaves are indicated in decision trees. But it can indeed add confusion.

[rth]

hope you don't mind I've just tried the "Request for reviewers" button

Don't hesitate to use that button on reviewers @cmarmo :) The code LGTM.

In numerical computing it
is not at all uncommon when working with positive ints to give negative
numbers special meanings. For instance, see how leaves are indicated in
decision trees.

Indeed, but I guess the issue here is that array index is not necessarily a positive integer, and that we are introducing a different meaning from what negative index commonly means in python. Another possibility for the sentinel could have been np.iinfo(np.int32).max () == 2147483647. Anyway any of these would likely be OK.

[cmarmo]

@rth, @thomasjpfan , after discussion with @glemaitre I've finally understood that self.drop_idx_ is set to None after fit so I can use it in the checks... :)
The code is a bit different wrt the version you approved, I have added a test for this particular situation.
Maybe you can find some time to check if you are ok anyhow? Thanks a lot for your patience.

[glemaitre]

LGTM apart of this small issue

[glemaitre]

Oh and we will need an entry in whats new:

Please add an entry to the change log at doc/whats_new/v0.23.rst under bug fixes. Like the other entries there, please reference this pull request with :issue: and credit yourself (and other contributors if applicable) with :user:

[glemaitre]

No need for the what's new the bug was only introduced in dev.

[cmarmo]

Someone available for merging? :) Thanks a lot!

[rth]

Thanks @cmarmo !