Repository navigation
Asterisks not encoded properly and result in error 32 "Could not authenticate you." #677
Description
Activity
I'm having the same issue when posting @ or # in my search terms. I can't figure out where the conflict is, so I had to revert to 5.13.0 for my app to run properly.
Reacted by Reema JosephI've found that the same results occur using the https://dev.twitter.com/rest/tools/console. This seems to be an issue with the API, not with the gem. Any thoughts?
Is there a work around for this? I thought I would be able to URL encode it manually if this gem doesn't do it but I don't get
*when I use%2ASorry to re-raise this topic, but I'm getting the same on master. I am going around temporarily by replacing asterisk by ＊. I also tried what dredenba suggested of using the url encoded version and it does not work.
Reacted by Sunil and /c/ympfhI think I've figured out the cause of "Could not authenticate you" with
*.https://github.com/sferik/twitter/blob/master/lib/twitter/rest/request.rb#L28 contains a call to
set_multipart_options!(request_method, options)which does@headers = Twitter::Headers.new(@client, @request_method, @uri, options).request_headers
This creates the oauth http headers, which signs the contents of the request, including the options.
When the request is sent
response = http_client.headers(@headers).public_send(@request_method, @uri.to_s, options_key => @options)
the http library encodes the form data, which is fine in most cases, but with
*(and likely some of the other specials in https://tools.ietf.org/html/rfc3986#section-2.2 the http library encodes the values, which changes the contents of the request.Since the request is now different, the twitter api rejects the request with a 401 Unauthorized.
Actually, after reading #888, it's the other way around, it's not being encoded.
It seems related to httprb/http#449 , which suggests a workaround.
This patch works for me.
diff --git a/lib/twitter/rest/request.rb b/lib/twitter/rest/request.rb index db34abbe..1c12c18a 100644 --- a/lib/twitter/rest/request.rb +++ b/lib/twitter/rest/request.rb @@ -32,8 +32,8 @@ module Twitter # @return [Array, Hash] def perform - options_key = @request_method == :get ? :params : :form - response = http_client.headers(@headers).public_send(@request_method, @uri.to_s, options_key => @options) + @uri.query_values = @options + response = http_client.headers(@headers).public_send(@request_method, @uri.to_s) response_body = response.body.empty? ? '' : symbolize_keys!(response.parse) response_headers = response.headers fail_or_return_response_body(response.code, response_body, response_headers)I'm also running into the same issue and can confirm that
URI#encode_www_forminside HTTP::FormData does not convert asterisks, whereas Twitter::Headers is relying on the gem simple_oauth which is usingURI::Parserand is encoding asterisks:URI::Parser.new.escape('*', /[^a-z0-9\-\.\_\~]/i) => "%2A". (https://github.com/laserlemon/simple_oauth/blob/be8a895c73a433dce47aa6ca37dd03a0a343fd55/lib/simple_oauth/header.rb#L29) Looks very much like there's an encoding incompatibility between these two.Hi,
The 2.1.1 version of
http-form_dataincludes a fix for this case discussed in httprb/form_data#22 and implemented in httprb/form_data@3bf2e83.It allows to override the
http-form_dataencoding method.The following snippet overrides the encoding to act like the
simple_oauthgem and thus fix the issue (you can put it inconfig/initializers/http_form_data.rbfor example):require "uri" # The encoder method of the http gem needs to be overriden because of the twitter gem. # Without that, there's an incompatibility between the simple_oauth gem which encodes asterisks and the http one which does not. # Cf. https://github.com/httprb/form_data/issues/22 and https://github.com/sferik/twitter/issues/677 HTTP::FormData::Urlencoded.encoder = lambda do |enum| unescaped_chars = /[^a-z0-9\-\.\_\~]/i enum.map do |k, v| if v.nil? ::URI::DEFAULT_PARSER.escape(k.to_s, unescaped_chars) elsif v.respond_to?(:to_ary) v.to_ary.map do |w| str = ::URI::DEFAULT_PARSER.escape(k.to_s, unescaped_chars) unless w.nil? str << '=' str << ::URI::DEFAULT_PARSER.escape(w.to_s, unescaped_chars) end end.join('&') else str = ::URI::DEFAULT_PARSER.escape(k.to_s, unescaped_chars) str << '=' str << ::URI::DEFAULT_PARSER.escape(v.to_s, unescaped_chars) end end.join('&') end
It's been a few weeks we're using this method and it seems fine so far. Be careful though, it can have some side effects if some of the other gems you're using depend on
http-form_dataand expect the original encoding way.Reviving this one to say it's still affecting the latest version
Reacted by Roope Salmi, Adam Dawkins, Ugurcan Kaya, Robert Rawlins, Ervin Kalemi, Philippe Le Rohellec, Scotty Weeks, Marc Köhlbrugge, Martin Edenhofer and Lukas Martini- added a commit that references this issue
on Jan 17, 2019 Running into this issue as well.
Does anyone know if it's just asterisks leading to problems, or other characters as well?
Reacted by Martin Edenhofer and Lukas MartiniFor me, the issue is only for asteriks. Other chars gets posted.
Reacted by Marc Köhlbrugge, Martin Edenhofer and Lukas MartiniLike @FabienChaynes said, the issue here is that oauth has different rules for escaping characters than the standard for application/x-www-form-urlencoded.
For the oauth signature, the * will be escaped, (* is missing from Reserved Characters here): https://developer.twitter.com/en/docs/basics/authentication/oauth-1-0a/percent-encoding-parameters
while for forms it is not: (0x2A is the *) https://www.w3.org/TR/2013/CR-html5-20130806/forms.html#url-encoded-form-dataThe http lib and simpleoauth both work as intended, Twitter just still wants x-www-form-urlencoded requests to be encoded by oauth rules, which doesn't work. (Actually it almost always works, except if your Message has an Asterisk...)
I came up with a solution that is basically the snippet from @FabienChaynes but this also checks if you are calling from inside the Twitter gem and decides based on that if it uses the default, or the decoding Twitter wants. That way we won't break the FormData Module for other libraries.
Also, I tried to make simpleoauth build the signature without encoding the *. Building the signature this way worked, but the Twitter API still didn't accept Tweets with * in them. It seems like the only way to get Twitter to accept tweets is to encode the request forms using oauth escaping rules.
I would have opened a pull request for this, but I don't really know where to put this, since I didn't find any obvious initialization that is done by the twitter gem. Maybe someone who is more familiar with the code has some idea about this?
require "uri" # The encoder method of the http gem needs to be overriden because of the twitter gem. # Without that, there's an incompatibility between the simple_oauth gem which encodes asterisks and the http one which does not. # Cf. https://github.com/httprb/form_data/issues/22 and https://github.com/sferik/twitter/issues/6# 77 # I added a check to see if this method has been called from inside the twitter gem, so that other libraries can still use the default behavior HTTP::FormData::Urlencoded.encoder = lambda do |enum, enc = nil| call_regex = Regexp.new( 'gems/twitter-[^/]+/lib/twitter/rest/request.rb:\d+:in `public_send\'') if caller.any? call_regex unescaped_chars = /[^a-z0-9\-\.\_\~]/i enum.map do |k, v| if v.nil? ::URI::DEFAULT_PARSER.escape(k.to_s, unescaped_chars) elsif v.respond_to?(:to_ary) v.to_ary.map do |w| str = ::URI::DEFAULT_PARSER.escape(k.to_s, unescaped_chars) unless w.nil? str << '=' str << ::URI::DEFAULT_PARSER.escape(w.to_s, unescaped_chars) end end.join('&') else str = ::URI::DEFAULT_PARSER.escape(k.to_s, unescaped_chars) str << '=' str << ::URI::DEFAULT_PARSER.escape(v.to_s, unescaped_chars) end end.join('&') else ::URI.encode_www_form(enum, enc) end end
Just for the record, I'm aware that including this snippet in the twitter lib isn't the best solution possible, even with the check for the calling function this could still break if some other lib also decides to override the default encoder, but I think this solution is still better than having this library not being able to send tweets with * in them. It's not great, but I think it's the next best thing to a clean solution.
IMHO a clean solution would require something like Faraday, where you can have separate instances of request handlers that can have their own rules for encoding to not interfere with other libraries. I only started looking at this repository yesterday, so I don't know why exactly the switch away from Faraday was made, but unless Twitter decides to change something about their API, using Faraday (or some other request library) is probably the only clean solution.
I’m an experienced developer, but I don’t speak Ruby. However, I went along with @Gordin in a long Twitter conversation (German though) while he dug into the issue and can confirm his conclusions.
Let me try to summarize what everyone has said:
- OAuth 1.0a (used by the Twitter API and implemented in this project using laserlemon/simple_oauth) requires the signature of a request to be computed over the request parameters encoded as described in RFC 3986, which states that
*is to be replaced by%2A. (Twitter’s API docs are very clear about this, too.) HTTP::FormData(used by this project to do the actual HTTP request) on the other hand encodes according to the rules of Ruby’sencode_www_form, which implements the HTML5 candidate recommendation, and for HTML5,*(0x2a) is not to be escaped anymore.- Twitter itself, when verifying the request, seems to compute the signature against the raw HTTP
POSTrequest, which will contain an unescaped asterisk produced byHTTP::FormData. But the signature we provide was made over a string containing an escaped asterisk as required by OAuth. Thus, the signature check fails and the tweet is rejected.- Twitter could of course adhere to Postel’s Law, decode the incoming request in a more liberal way, and then compute its own signature according to OAuth’s rules by re-escaping the data in the RFC 3986 way, but alas, they don’t.
None of the libraries that this project is using is really at fault here: They all do their job correctly. However, the OAuth and the HTTP code escape the parameters according to different standards, which then causes the issue. That makes this library the correct place to fix it, because it is supposed to connect all the threads correctly, and it doesn’t. (Sorry 😉)
Overriding the way in which the parameters are escaped in the HTTP request seems to be the only way to do this, and apart from switching to a different HTTP library, the snippet suggested by @Gordin seems to me to be the best solution.
Reacted by Marvin, Kazune Takahashi and Lukas Martini- OAuth 1.0a (used by the Twitter API and implemented in this project using laserlemon/simple_oauth) requires the signature of a request to be computed over the request parameters encoded as described in RFC 3986, which states that
If the
http-form_datagem would allow you to customize encoding on a per-instance basis and we could pass in an encoder through thehttpgem this could solve the problem without having to usecallerto determine what's making the request. I've opened httprb/form_data#29 to see if that's a possible direction.Reacted by Tim Weber and Renato "Lond" CerqueiraWith updates from httprb/http#599 and httprb/form_data#29, the
httpgem can now accept an object with a custom encoder as ofv4.4. I just opened #969 to fix this encoding issue. If you all wouldn't mind testing this with your projects and letting me know if it works, that would be greatly appreciated. I'll be doing the same.
Twitter-text (1.11.0) says that tweets with asterisks are valid. And I am also allowed to post them to Twitter directly. However, when posted through the Twitter gem (6.0.0) I receive the error: "Could not authenticate you." (code 32)
This forum post indicates that asterisks need to be URL encoded to be valid, so shouldn't the Twitter gem be handling this for me?
https://twittercommunity.com/t/asterisk/6343