BUG: np.str_ and np.bytes_ object have a confusing __repr__ and __str__

>>> s = np.str_("\0\0")
>>> list(s)
['\0', '\0']
>>> len(s)
2
>>> s
''  # wat

>>> b = np.bytes_(2)
>>> list(b)
[0, 0]
>>> len(b)
2
>>> b
b''  # wat

Caused by

numpy/numpy/core/src/multiarray/scalartypes.c.src

Lines 389 to 425 in f15026c

    
           /* 
        
            * over-ride repr and str of array-scalar strings and unicode to 
        
            * remove NULL bytes and then call the corresponding functions 
        
            * of string and unicode. 
        
            */ 
        
           /**begin repeat 
        
            * #name = string*2,unicode*2# 
        
            * #form = (repr,str)*2# 
        
            * #Name = String*2,Unicode*2# 
        
            * #NAME = STRING*2,UNICODE*2# 
        
            * #extra = AndSize*2,,# 
        
            * #type = npy_char*2, Py_UNICODE*2# 
        
            */ 
        
           static PyObject * 
        
           @name@type_@form@(PyObject *self) 
        
           { 
        
               const @type@ *dptr, *ip; 
        
               int len; 
        
               PyObject *new; 
        
               PyObject *ret; 
        
               ip = dptr = Py@Name@_AS_@NAME@(self); 
        
               len = Py@Name@_GET_SIZE(self); 
        
               dptr += len-1; 
        
               while(len > 0 && *dptr-- == 0) { 
        
                   len--; 
        
               } 
        
               new = Py@Name@_From@Name@@extra@(ip, len); 
        
               if (new == NULL) { 
        
                   return PyUString_FromString(""); 
        
               } 
        
               ret = Py@Name@_Type.tp_@form@(new); 
        
               Py_DECREF(new); 
        
               return ret; 
        
           } 
        
           /**end repeat**/

It looks to me like we should just remove our overloads of these functions, they don't make any sense.

Is it like this to be consistent with the overall stripping of trailing NULs in S and U dtypes in arrays? It's not just string repr, but also x = np.array([b'a\0\0\0']); assert type(x[0]) is np.bytes_; assert x[0] == b'a' and probably other things too.

If that's the intent, we should be stripping the nulls upon construction, not upon printing. As a more obviously broken case, consider:

>>> s = np.str_("ab\0")
>>> s
"ab"
>>> s[-1:]
""

I think this only affects scalars constructed directly, not those extracted from arrays.

I'm not sure whether np.bytes_(b'a\0') == b'a' is actually true, even though it looks like it is when you print

Indeed, len(x[0]) == 1, so item extraction seems to generate NUL-stripped string scalars. So maybe the reprs indeed don't make sense.

Yeah, the NULL stripping is always an issue we have, and something we cannot get around. I think the mental model of the NULLs being stripped when the NumPy scalar is created is the best thing we can probably do. We could think about adding a warning when null padded strings/bytes are put into a NumPy array, since they will not roundtrip correctly anyway...

So I think after removing these overloads, our options are to have the numpy subclasses:

Behave exactly like the builtin base class, other than having the .dtype attribute (ie, do nothing else)
Strip trailing NULLs on construction. Note this will cause things like len(np.str_("a\0b")[:2]) == 1, which is perhaps surprising
Strip trailing NULLs on construction but warn
Warn if trailing NULLs are present, but keep them

A variant of 4 would be to pick 1, and instead change UNICODE_setitem to warn when trailing nulls are stripped, which might be more user-friendly, and would catch the behavior whether numpy scalars are used or not.

I am for Option 3.

Any option should have the same result as np.array(string, dtype="S")[()] and np.array(str, "S") and np.string_(str) should be as similar as possible (at the moment at least, due to PyArray_Return).

Since it can be a bug for someone who needs the zero bytes, I think a warning makes sense. The only question is whether the warning for some reason is prohibitively noisy. I do not expect that. It could point out that void can sometimes be a way to spell bytes of fixed length.

I'm leaning towards the option 4 variant. The more I think about it, 3 violates a lot of assumptions that being a subclass of str would imply.

Consider:

def replace_ext(s):
    a, b = s.split('.',1)
    return a + ".npy"

If passed a np.str_ with your proposed changes, this will remove inner nulls from "a\0.ext"

True, although that code will probably give the warning (although if you have many of that you would not notice). Its true from a subclassing perspective...
It seems we either have to sacrifice consistent subclassing or consistency with arrays? Since this is NumPy scalar, which is almost an array, I think I still lean a bit towards 3. But basically there is no good solution and either 3. or 4. seem better than the current state to me.

@seberg

I'm with @seberg - personally I'd choose consistency with arrays, but either option that emits a warning is reasonable.

Navigation Menu

Repository navigation

BUG: np.str_ and np.bytes_ object have a confusing repr and str #15477

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	/*
	* over-ride repr and str of array-scalar strings and unicode to
	* remove NULL bytes and then call the corresponding functions
	* of string and unicode.
	*/

	/**begin repeat
	* #name = string2,unicode2#
	* #form = (repr,str)*2#
	* #Name = String2,Unicode2#
	* #NAME = STRING2,UNICODE2#
	* #extra = AndSize*2,,#
	* #type = npy_char2, Py_UNICODE2#
	*/
	static PyObject *
	@name@type_@form@(PyObject *self)
	{
	const @type@ dptr, ip;
	int len;
	PyObject *new;
	PyObject *ret;

	ip = dptr = Py@Name@_AS_@NAME@(self);
	len = Py@Name@_GET_SIZE(self);
	dptr += len-1;
	while(len > 0 && *dptr-- == 0) {
	len--;
	}
	new = Py@Name@_From@Name@@extra@(ip, len);
	if (new == NULL) {
	return PyUString_FromString("");
	}
	ret = Py@Name@_Type.tp_@form@(new);
	Py_DECREF(new);
	return ret;
	}
	/end repeat/

BUG: np.str_ and np.bytes_ object have a confusing __repr__ and __str__ #15477

Description

Activity

pv commented on Jan 30, 2020

eric-wieser commented on Jan 30, 2020

pv commented on Jan 30, 2020

seberg commented on Jan 30, 2020

eric-wieser commented on Feb 1, 2020

seberg commented on Feb 1, 2020

eric-wieser commented on Feb 1, 2020

seberg commented on Feb 1, 2020

Zac-HD commented on Feb 2, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

BUG: np.str_ and np.bytes_ object have a confusing repr and str #15477