欢迎各位兄弟 发布技术文章

这里的技术是共享的

You are here

Disallowed Unicode code point in CTOOLS_PREG_CLASS_ALNUM 有大用 有大大用

Constant CTOOLS_PREG_CLASS_ALNUM in includes/cleanstring.inc contains \x{d800}- which is ill-formed code point. PHP issues this warning:

Warning: preg_match(): Compilation failed: disallowed Unicode code point (>= 0xd800 && <= 0xdfff) at offset 1811 in ctools_cleanstring() (line 157 of /srv/http/XXXX/www/sites/all/modules/contrib/ctools/includes/cleanstring.inc).                    

Easily reproducable by running this:

define('CTOOLS_PREG_CLASS_ALNUM',
'\x{0}-\x{2f}\x{3a}-\x{40}\x{5b}-\x{60}\x{7b}-\x{bf}\x{d7}\x{f7}\x{2b0}-' .
'\x{385}\x{387}\x{3f6}\x{482}-\x{489}\x{559}-\x{55f}\x{589}-\x{5c7}\x{5f3}-' .
'\x{61f}\x{640}\x{64b}-\x{65e}\x{66a}-\x{66d}\x{670}\x{6d4}\x{6d6}-\x{6ed}' .
'\x{6fd}\x{6fe}\x{700}-\x{70f}\x{711}\x{730}-\x{74a}\x{7a6}-\x{7b0}\x{901}-' .
'\x{903}\x{93c}\x{93e}-\x{94d}\x{951}-\x{954}\x{962}-\x{965}\x{970}\x{981}-' .
'\x{983}\x{9bc}\x{9be}-\x{9cd}\x{9d7}\x{9e2}\x{9e3}\x{9f2}-\x{a03}\x{a3c}-' .
'\x{a4d}\x{a70}\x{a71}\x{a81}-\x{a83}\x{abc}\x{abe}-\x{acd}\x{ae2}\x{ae3}' .
'\x{af1}-\x{b03}\x{b3c}\x{b3e}-\x{b57}\x{b70}\x{b82}\x{bbe}-\x{bd7}\x{bf0}-' .
'\x{c03}\x{c3e}-\x{c56}\x{c82}\x{c83}\x{cbc}\x{cbe}-\x{cd6}\x{d02}\x{d03}' .
'\x{d3e}-\x{d57}\x{d82}\x{d83}\x{dca}-\x{df4}\x{e31}\x{e34}-\x{e3f}\x{e46}-' .
'\x{e4f}\x{e5a}\x{e5b}\x{eb1}\x{eb4}-\x{ebc}\x{ec6}-\x{ecd}\x{f01}-\x{f1f}' .
'\x{f2a}-\x{f3f}\x{f71}-\x{f87}\x{f90}-\x{fd1}\x{102c}-\x{1039}\x{104a}-' .
'\x{104f}\x{1056}-\x{1059}\x{10fb}\x{10fc}\x{135f}-\x{137c}\x{1390}-\x{1399}' .
'\x{166d}\x{166e}\x{1680}\x{169b}\x{169c}\x{16eb}-\x{16f0}\x{1712}-\x{1714}' .
'\x{1732}-\x{1736}\x{1752}\x{1753}\x{1772}\x{1773}\x{17b4}-\x{17db}\x{17dd}' .
'\x{17f0}-\x{180e}\x{1843}\x{18a9}\x{1920}-\x{1945}\x{19b0}-\x{19c0}\x{19c8}' .
'\x{19c9}\x{19de}-\x{19ff}\x{1a17}-\x{1a1f}\x{1d2c}-\x{1d61}\x{1d78}\x{1d9b}-' .
'\x{1dc3}\x{1fbd}\x{1fbf}-\x{1fc1}\x{1fcd}-\x{1fcf}\x{1fdd}-\x{1fdf}\x{1fed}-' .
'\x{1fef}\x{1ffd}-\x{2070}\x{2074}-\x{207e}\x{2080}-\x{2101}\x{2103}-\x{2106}' .
'\x{2108}\x{2109}\x{2114}\x{2116}-\x{2118}\x{211e}-\x{2123}\x{2125}\x{2127}' .
'\x{2129}\x{212e}\x{2132}\x{213a}\x{213b}\x{2140}-\x{2144}\x{214a}-\x{2b13}' .
'\x{2ce5}-\x{2cff}\x{2d6f}\x{2e00}-\x{3005}\x{3007}-\x{303b}\x{303d}-\x{303f}' .
'\x{3099}-\x{309e}\x{30a0}\x{30fb}-\x{30fe}\x{3190}-\x{319f}\x{31c0}-\x{31cf}' .
'\x{3200}-\x{33ff}\x{4dc0}-\x{4dff}\x{a015}\x{a490}-\x{a716}\x{a802}\x{a806}' .
'\x{a80b}\x{a823}-\x{a82b}\x{d800}-\x{f8ff}\x{fb1e}\x{fb29}\x{fd3e}\x{fd3f}' .
'\x{fdfc}-\x{fe6b}\x{feff}-\x{ff0f}\x{ff1a}-\x{ff20}\x{ff3b}-\x{ff40}\x{ff5b}-' .
'\x{ff65}\x{ff70}\x{ff9e}\x{ff9f}\x{ffe0}-\x{fffd}');

if (preg_match('/^[^' . CTOOLS_PREG_CLASS_ALNUM . ']+$/uD', '-')) {
  echo 'Warning issued';
}
                   

Patch to be posted in comment.

Files: 
CommentFileSizeAuthor
#37                                ctools-1444006-disallowed-unicode-D6-37.patch                                2.49 KBgeneralconsensus                                
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch ctools-1444006-disallowed-unicode-D6-37.patch. Unable to apply patch. See the log in the details link for more information. View                                    
#18                                ctools-1444006-disallowed-unicode-code-point-18.patch                                1.24 KBgdl                                
PASSED: [[SimpleTest]]: [MySQL] 0 pass(es). View                                    
#7                                ctools-1444006-7-bad-unicode.patch                                2.35 KBkotnik                                
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch ctools-1444006-7-bad-unicode_0.patch. Unable to apply patch. See the log in the details link for more information. View                                    
#6                                remove-annoying-warnings-about-unicode-D6.patch                                726 bytesMarc Angles                                
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch remove-annoying-warnings-about-unicode-D6.patch. Unable to apply patch. See the log in the details link for more information. View                                    
#1                                ctools-1444006-1-removed-bad-unicode.patch                                1.21 KBkotnik                                
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch ctools-1444006-1-removed-bad-unicode.patch. Unable to apply patch. See the log in the details link for more information. View                                    

Comments

kotnik’s picture        

 

Status:Active» Needs review
FileSize
1.21 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch ctools-1444006-1-removed-bad-unicode.patch. Unable to apply patch. See the log in the details link for more information. View                                        

Patch with fix attached.

merlinofchaos’s picture        

 

Status:Needs review» Fixed

Nice! And a properly formatted commit message which is rare! Thank you.

activelink’s picture        

 

This bug is likely caused by pcre 8.30+, most Linuxes still use older versions, but FreeBSD has 8.30_1 currently. But this version of pcre will be on Linux soon - it's developed on FreeBSD.

It breaks core search too in D6/7.

Ctools has it because it has code in cleanstring.inc that is copied from includes/unicode.inc (D7 core) or modules/search/search.module in D6.

It's examined in some depth here:
Invalid Unicode code range in PREG_CLASS_UNICODE_WORD_BOUNDARY fails with PCRE 8.30                        

There are patches for D6 and D7, and these work also if adapted for ctools' unicode.inc

These patches are slightly different to the one above. They replace the D800 parameter with E800 and this works. The patch above removes it entirely. I don't understand this code or what it does, but it seems like the patches that I'm referencing are more evolved than the one given on this page (which removes this value rather than replacing it). Maybe someone who understands its function might want to look at it?

kotnik’s picture        

 

Yes, this is just the same issue as one described in #1446372: Invalid Unicode code range in PREG_CLASS_UNICODE_WORD_BOUNDARY fails with PCRE 8.30 (pcre > 8.30, observable in FreeBSD and Arch, probably Gentoo too).

As a matter of fact, CTOOLS_PREG_CLASS_ALNUM (changed in patch from #1) is copied from search.module's PREG_CLASS_SEARCH_EXCLUDE.

Removing d800 is quite safe, since CTOOLS_PREG_CLASS_ALNUM is used to escape separator if it's not alphanumeric.

 

Status:Fixed» Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

activelink’s picture        

 

FileSize
726 bytes
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch remove-annoying-warnings-about-unicode-D6.patch. Unable to apply patch. See the log in the details link for more information. View                                        

Hi, I really don't know if this patch is right, but for the moment there is no more messages on my websites (This is D6.25)

kotnik’s picture        

 

Priority:Normal» Critical
Status:Closed (fixed)» Needs review
FileSize
2.35 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch ctools-1444006-7-bad-unicode_0.patch. Unable to apply patch. See the log in the details link for more information. View                                        
1.2 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch ctools-1444006-7-bad-unicode.patch. Unable to apply patch. See the log in the details link for more information. View                                        

Updated to be in sync with #1446372: Invalid Unicode code range in PREG_CLASS_UNICODE_WORD_BOUNDARY fails with PCRE 8.30.

As noted in 1446372's comment #16 it's better to scan Unicode private use instead of surrogate area, then to ignore it.

This code has been merged in Drupal core 8.x and 7.x (patch waiting to be ported to 6.x), and I think ctools should do the same.

PS. @Marc Angles: yes, that's the same fix that'll be included in next 6.x core.

lucascaro’s picture        

 

I've just manually replaced

'\x{a80b}\x{a823}-\x{a82b}\x{d800}-\x{f8ff}\x{fb1e}\x{fb29}\x{fd3e}\x{fd3f}'.
                       

with
'\x{a80b}\x{a823}-\x{a82b}\x{e000}-\x{f8ff}\x{fb1e}\x{fb29}\x{fd3e}\x{fd3f}'.                        

as in #7 and it fixed the error messages in 6.x-1.8+50-dev

Since the first half of the patch seems to be already committed, that last change is all that is needed to remove the message (at least using my configuration).

RTBC, anyone?

kotnik’s picture        

 

Status:Needs review» Reviewed & tested by the community

The exact same fix as in #7 has already been commited in D6, D7, and D8. Marking it as RTBC.

lucascaro’s picture        

 

are they committed to dev? awesome! also, maybe this should be marked as fixed?
thanks!

kotnik’s picture        

 

@lucascaro:

No, patch from #7 needs to be commited to ctools, since the exact same fix was applied to Drupal 6, 7 and 8.

lucascaro’s picture        

 

ooh, nice. This is starting to trigger error messages in all of my projects, so I'd love it to be committed. How can I help making it happen?

tim.plunkett’s picture        

 

Assigned:Unassigned» tim.plunkett                                    

For tonight.

lucascaro’s picture        

 

Assigned:tim.plunkett                                    » Unassigned

awesome!

tim.plunkett’s picture        

 

Status:Reviewed & tested by the community» Fixed

Oh wow the format-patch threw me off there. This is much more straightforward then I thought.

Also, why'd you unassign me? I almost couldn't find the issue.

http://drupalcode.org/project/ctools.git/commit/a1c5c9b                        

lucascaro’s picture        

 

og god, sorry about that. I don't know why that happened, I didn't touch the assigned box. :S

 

Status:Fixed» Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

activelink’s picture        

 

Version:7.x-1.x-dev» 6.x-1.x-dev
Status:Closed (fixed)» Needs review
FileSize
1.24 KB
PASSED: [[SimpleTest]]: [MySQL] 0 pass(es). View                                        

Sorry to reopen, but it looks to me like this didn't make it into 6.x-1.x-dev.

Attached is a patch for 6.x-1.x-dev made by copying PREG_CLASS_SEARCH_EXCLUDE from search.module from Drupal 6.26. Works for me, but not thoroughly tested.

Best,
-G

hansfn’s picture        

 

I agree that this seems to be unfixed still.

@gdl: Why the difference between your patch and the one in #1697538: Unicode compilation fails because of invalid code range in CTOOLS_PREG_CLASS_ALNUM?

activelink’s picture        

 

It looks like the patch in #1697538 changes just a single code point, while mine changes the list of code points in CTools to match the code points from the core search module. I guess it's up the the developers to determine which approach to take. I don't have an informed opinion about which is better - I just want to see the issue resolved!

Cheers,
-G

jhedstrom’s picture        

 

Status:Needs review» Reviewed & tested by the community

Fix in #18 gets rid of the warnings in 6.x-1.x.

activelink’s picture        

 

The #18 patch works here, with pcre package version 7.8 and ctools 6.x-1.10+0-dev.

activelink’s picture        

 

confirmin, patch from #18 fixes the issues you get when using PHP > 5.3.18 . Applying did not succeed though. Thanks!

*removed since it up there already*

activelink’s picture        

 

Just upgraded to 6.x-1.10, and found the error from the OP was again occurring, #8 again solves the problem.

Why is this not changed in the module core?

neclimdul’s picture        

 

Status:Reviewed & tested by the community» Closed (fixed)

Thanks guys. Tested and applied #1697538-2: Unicode compilation fails because of invalid code range in CTOOLS_PREG_CLASS_ALNUM since it brings the regex into a state matching the 7.x-1.x branch it seemed more reasonable. If we need the extra change here can you reroll with an explanation?

activelink’s picture        

 

hello
i am not professional in Drupal core settings

and this error appear for me too

warning: preg_match(): Compilation failed: disallowed Unicode code point (>= 0xd800 && <= 0xdfff) at offset 1811 in C:\inetpub\vhosts\xxxxx\httpdocs\sites\all\modules\ctools\includes\cleanstring.inc on line 157.

how can i fix this problem pleas ????

activelink’s picture        

 

#8 Work for me :D

I've just manually replaced

'\x{a80b}\x{a823}-\x{a82b}\x{d800}-\x{f8ff}\x{fb1e}\x{fb29}\x{fd3e}\x{fd3f}'.

with
'\x{a80b}\x{a823}-\x{a82b}\x{e000}-\x{f8ff}\x{fb1e}\x{fb29}\x{fd3e}\x{fd3f}'.

crutch’s picture        

 

this is happening more often after I search and search, then post, I find the issue when searching google on drupal. I could not find this issue in Drupal search.

possible duplicate: http://drupal.org/node/1878284                        

This seems still unfixed in the module.

activelink’s picture        

 

I solved it doing this. I TRUNCATE the following tables: cache_page, acceslog, cache_menu and cache_filter.

I think that commen #27 don't work for me, but i set the line like beto_beto say:

'\x{a80b}\x{a823}-\x{a82b}\x{e000}-\x{f8ff}\x{fb1e}\x{fb29}\x{fd3e}\x{fd3f}'.

gnuget’s picture        

 

I've the same bug and also the patch from #18 fixed it.

activelink’s picture        

 

I just ran into this problem for this first time today (6.x-1.10).

#8 solved the problem.

activelink’s picture        

 

Version:6.x-1.x-dev» 6.x-1.10

I'm on 6.x-1.10, running PHP 5.3.21, and running Drupal 6.28. Anyone know if #18 or #8 is the better route to take?

activelink’s picture        

 

I've used the approach from #8 and that's solved the problem for me.

activelink’s picture        

 

I was getting

warning: preg_match() [function.preg-match]: Compilation failed: disallowed Unicode code point (>= 0xd800 && <= 0xdfff) at offset 1811 in /var/www/mysite/public_html/sites/all/modules/ctools/includes/cleanstring.inc on line 157

a number of times after upgrading my server from

PHP 5.2.17-0.dotdeb.0
MySQL 5.1.56

to

PHP 5.3.21-1~dotdeb.0
MySQL 5.1.66

on a Drupal 6.28 site running cTools 6.x-1.10, but #8 solved the problem. Not sure why this hasn’t been committed!

kmare’s picture        

 

Same as Grabby for me, #8 fixed the problem on my drupal6 installations. I really believe it should be commited.

activelink’s picture        

 

Thanks! I had the same issue as #34: the error appeared after our servers were upgraded to PHP5.3.x. The simple fix in #8 resolved the problem.

generalconsensus’s picture        

 

FileSize
2.49 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch ctools-1444006-disallowed-unicode-D6-37.patch. Unable to apply patch. See the log in the details link for more information. View                                        

The fix from #8 works but doesn't affect search.module. This patch incorporates #8 with a fix for the search module using the same code.

Works for me on:
FREEBSD
Drupal 6.16
PHP 5.3.19
APACHE 2.2.23

dadderley’s picture        

 

Another vote for solution in #8

activelink’s picture        

 

Confirm that patch in #18 worked - CTools 6.x-1.10, Drupal 6.28, PHP 5.3.19, Debian 6.0.6

activelink’s picture        

 

Status:Closed (fixed)» Reviewed & tested by the community

#18 worked very well for my installation :
CTools 6.x-1.10
PHP 5.3.23
Drupal 6.28

I updated to "tested by the community" because it's enough trivial to include it into git.

 

Status:Reviewed & tested by the community» Needs work

The last submitted patch, ctools-1444006-disallowed-unicode-D6-37.patch, failed testing.

merlinofchaos’s picture        

 

Status:Needs work» Closed (fixed)

As per #25, this was marked fixed; neclimdul committed the patch. Why are there 17 comments after that asking for this to be committed?

The reason it failed testing is the patch, which is already committed, doesn't apply.

Do people actually read issues before they start in? Because it seems like a giant waste of time to have so many people asking why a fix, which is committed, hasn't been committed. Because it's been committed.

hansfn’s picture        

 

Status:Closed (fixed)» Active

There are 17 comments because, as I said already in comment 19, the fix was only committed to the 7.x-1.x branch, not the 6.x-1.x branch. Hence, the bug is still present on 6.x-1.10... This is clearly frustrating for people using Drupal 6. (The fix is almost one year old.)                        

OK, merlin, you are right. The fix was applied to the 6.x-1.x branch - ref this commit. So the only problem is that there hasn't been a new release of Ctools since 6.x-1.10.

hansfn’s picture        

 

Version:6.x-1.10» 6.x-1.x-dev
Status:Active» Closed (fixed)

Updating version info since this isn't fixed for 6.x-1.10, only in 6.x-1.x-dev.

hansfn’s picture        

 

Version:6.x-1.x-dev» 6.x-1.10

Sorry about the noise. The bug, which is fixed, was in version 6.x-1.10. Some times I'm just that stupid.

Anybody’s picture        

 

This is a quite important issue. Wouldn't it make sense to release a new stable version 2 month after that? We all have to switch to .dev else.

KarlShea’s picture        

 

Status:Closed (fixed)» Active

I have 6.x-1.10 installed and the commit you reference that fixes the bug has not been applied.

cleanstring.inc line 61 is:

'\x{a80b}\x{a823}-\x{a82b}\x{d800}-\x{f8ff}\x{fb1e}\x{fb29}\x{fd3e}\x{fd3f}'.                        

d800 should be e000

GiorgosK’s picture        

 

Status:Active» Reviewed & tested by the community

Had same problem with 6.x-1.10

#18 patch solves this problem
#37 includes a patch for core search module that should be posted to core queue

merlinofchaos’s picture        

 

Status:Reviewed & tested by the community» Closed (fixed)

As per #42, this was committed. It was committed after 6.x-1.10 was released. Please pay attention when changing issue statuses.

And I'm sorry there hasn't been a new release, but releases for Drupal 6 are not high on my priority list.

KarlShea’s picture        

 

Uh, I was paying attention. #45 indicates that it was in 6.x-1.10. Which it isn't.

ifrik’s picture        

 

patch #18 worked for me

activelink’s picture        

 

patch #18 worked for me too.. :)

activelink’s picture        

 

Issue summary:View changes                                    

:)

activelink’s picture        

 

#8 Work for ME

this error was appear to me

Warning: preg_match(): Compilation failed: disallowed Unicode code point (>= 0xd800 && <= 0xdfff) at offset 1811 in ctools_cleanstring() (line 157 of /srv/http/XXXX/www/sites/all/modules/contrib/ctools/includes/cleanstring.inc).

Solution
on
sites/all/modules/contrib/ctools/includes/cleanstring.inc

I've just manually replaced
'\x{a80b}\x{a823}-\x{a82b}\x{d800}-\x{f8ff}\x{fb1e}\x{fb29}\x{fd3e}\x{fd3f}'.

with
'\x{a80b}\x{a823}-\x{a82b}\x{e000}-\x{f8ff}\x{fb1e}\x{fb29}\x{fd3e}\x{fd3f}'.

Thanks

activelink’s picture        

 

me too

activelink’s picture        

 

thanks @gdl (comment #18) for the D6 patch. Works great.

activelink’s picture        

 

Related issues: +#1697538: Unicode compilation fails because of invalid code range in CTOOLS_PREG_CLASS_ALNUM                                    

@50 and for general clarification : #45 indicates that the bug was in 6.x-1.10 (which was released Nov 14, 2012) and since the fix was committed on Nov 29, 2012 it only appears in subsequent releases, i.e. 6.x-1.11 released Jan 12, 2014 and any later releases that there may be.

davidwbarratt’s picture        

 

Version:6.x-1.10» 7.x-1.0-rc1
Status:Closed (fixed)» Active

I'm receiving the same/similar error on 7.x-1.0-rc1:

Warning: preg_match(): Compilation failed: disallowed Unicode code point (>= 0xd800 && <= 0xdfff) at offset 1811 in ctools_cleanstring() (line 157 of includes/cleanstring.inc).
                       

Any ideas?

 

  • merlinofchaos committed 8fe7bb5 on 8.x-2.x authored by kotnik

    Issue #1444006 by kotnik: Removed disallowed Unicode code point.
    
    

                                       


  • tim.plunkett committed a1c5c9b on 8.x-2.x authored by kotnik

    Issue #1444006 by kotnik, lucascaro: Fixed disallowed Unicode code point...

                                       


DamienMcKenna’s picture        

 

Version:7.x-1.0-rc1» 6.x-1.10
Status:Active» Closed (fixed)
Parent issue: » #1774796: Plan for Panels v6.x-3.11 release (PHP 5.4 fix)                                    

@davidwbarratt: Please open a new issue for the D7 problem if it persists in the latest D7 release.


来自  https://www.drupal.org/node/1444006


普通分类: