PDFFont exception

The enclosed PDF
CompilationMapLegend.pdf
threw an exception in PDFFont line 140 with cid=31.  Once the exception happens I can no longer process any information on that page.  It would be nice to find a way to throw an exception to notify there was a bad character, but then ignore that character and continue processing.  I worked around the problem by replacing the re-raise with returning self.cid2unicode[32] (space).
This document had multiple occurrences of bad characters, all cid=31.  They all appeared to happen at the start of a string.  The first occurrence was on PDF page 3 (page_num=2).  I do not understand CMap or CID.  ASCII 31 is "unit separator".  In ASCII this appears to be a "divider" for plain text and probably could be ignored in my context.  My assumption is these bad characters are really part of the text, but another possibility is there is a parsing problem.

This issue requires a bit of thought. I am thinking that a good fix for this would be to include an option during parsing that would allow "bad characters".
Perhaps something like this
PDFInterpreter("<filename>", ignore_bad_chars=True)

What this would do is whenever a PDFUnicodeNotDefined should be thrown, I will instead return a null character '', and keep the parsing process going.
Let me know what you think.
As for the mapping issue,  not sure what to say, will need to explore it more deeply.

Thanks for thinking about this.  I agree it requires some thought.  I think your proposed solution is good.  It preserves the simple interface of pdfmajor.
When I understood my pdfmajor exceptions were due to bad characters my first thought was “where are the bad characters” and “are they important”.  I first substituted a  and I still could not figure out where the bad characters were.  Next I substituted a less common character like ‘Z’ and compared outputs.  In a big document knowing roughly where the bad characters are makes it easier to manually check if there is important data loss.  In my case, after opening a PDF viewer and looking the location of these bad characters, I believe that ignoring these characters would be fine.  My guess is a “Normal” person may not care where the bad characters are.  I always have the option to modify the source for whatever troubleshooting I want to to.
I thought about other methods that would allow more troubleshooting of a PDF with unsupported characters but I think these would be too complicated.  For instance having an interpreter parameter that allows the user to assign a bad character handler.  “def bad_character_handler(cid) -> unicode”.  This would give the user the ability to see the bad cid and assign any character, or ‘’, as the substitute.  Even without knowing what a cid is the user would have the ability to count how many there are.  If only 1 character is bad no big deal.  If 50% are bad big deal.
Another option that is more like your proposal is:
PDFInterpreter("”, bad_char_substitute=None)
None : throws exception on bad character as today
‘’ : empty string ignores bad character
‘’ :  substitutes for bad character
This would allow me to do what I did to troubleshoot without having to modify the pdfmajor source.  Would other people use this extra functionality?  Probably only a few.
I do not understand CMaps or CID well.  Are there some CID with values 0..31 that are safe to always ignore?  That would fix my problem and maybe help others avoid any exception.  It would not fix problems with other bad character stopping processing on on PDF page.
Do not be too influenced by me.  I am an amateur hack.  I really like pdfmajor and looking through the source it looks well designed.  I trust your judgement more than my own.  Do what you think is best.  I liked the package and just wanted to provide some feedback for you to consider.  I am available if you want to use me as a sounding board other concepts.
BTW.  PDFMajor was the only PDF package I found that could handle color and it was the simplest to use.  Once I worked around the bad characters it did exactly what I needed (get the location of text and rects, get the fill color of rects).  Unfortunately the very first PDF I picked as a test case had problems.
-brett

On Dec 7, 2019, at 10:34 PM, Ari Sosnovsky<EMAIL_ADDRESS>wrote:
This issue requires a bit of thought. I am thinking that a good fix for this would be to include an option during parsing that would allow "bad characters".
Perhaps something like this
PDFInterpreter("", ignore_bad_chars=True)
What this would do is whenever a PDFUnicodeNotDefined should be thrown, I will instead return a null character '', and keep the parsing process going.
Let me know what you think.
As for the mapping issue, not sure what to say, will need to explore it more deeply.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/asosnovsky/pdfmajor/issues/2?email_source=notifications&email_token=ANKCUCOEZ4NWI7I3YY7YOQTQXR2OZA5CNFSM4JW3NT62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGGVXNY#issuecomment-562912183, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANKCUCMSSZRD5XSDG7C7HRLQXR2OZANCNFSM4JW3NT6Q.

Thanks for the kind words.
I like the idea of adding a counter to the number of wrong characters. I could implement an error-state and keep track of the number of bad-characters (as well as other errors that could propagate). Then i could raise a warning if anything has been logged to the error-state (plus make it accessible via PDFInterpreter.errors, where I could store some contextual information about the bad-characters, like where they were found).
So it would likely be a class like this
class ErrorState:
     bad_chars: List[BadChar]

class BadChar(NamedTuple):
     cid: int
     lead_chars: List[str]
     follow_chars: List[str]
     page_num: int

What do you think? In the meanwhile I have implemented ignore_bad_chars flag, so that you can run this clearly, but it only replaces the bad-characters with "".
Some context, this library is actually just a refactoring project of the pdfminer.six project. Specifically, because I needed access to color and reliable rect positioning information (in the way I also took the liberty on modding the structure of repo, and added a bit of a speed-boost and more reliable html convertion). Given so, my understanding of CMap might not be any better than yours. The most I know is that it is a mapping of characters within the document. So if a cid is not found, that would mean that either;

I did not port the implementation well enough
the original writers in pdfminer made a mistake
the tool you used to make the pdf errored while writing the table and some of rows are missing (or perhaps the character itself was written wrong)
the pdf document was corrupted at one point

I think your proposed solution is good.  I think it provides everything to troubleshoot but is transparent if you do not want to troubleshoot.
I am not sure I understand the type “List[str]” for lead_chars and follow_chars.  Would the str be limited to the text within an LTCharBlock?  My initial thought was this could be just str but I am not sure what text you intend to include.  Is the reason for List because you would break up the text with a LTCharBlock into chunks delimited by a bad chars?
I am not sure how difficult this would be to implement but consider another option:
class BadChar(NamedTuple):
cid: int
page_num: int
char_block: LTCharBlock (or maybe char_block_text: str)
position: int (index into charblock text)
If it is problematic or inefficient for the parser to retain every instance of LTCharBlock with errors you could used the alternate of just str.  Feel free to modify my variable names.
A few comments about your thinking if cid is not found.
I ran the PDF I included through "pdfminer 20191125” and I did not encounter any exceptions or obvious errors.  I do not know how this differs from pdfminer.six.  At the time I didn’t realize there were questionable characters in the PDF and I never went back to try compare text to see if it was simply ignoring those characters.  After I concluded it did not support colors I never spent much time with it.
If you look at the PDF metadata it appears to be produced by a popular PDF product.  This could certainly have bugs but if so many PDF’s might have this.
PDF Producer: Acrobat Distiller 7.0 (Windows)
Content Creator: PScript5.dll Version 5.2.2
The metadata also has a Title “ClassSymbolReport”.  This hints at a script.  The data may have been coming from a GIS program or database program (Access, MySQL, etc) and the bad chars might be some byproduct of the script or data source.  Other applications like macOS Preview or Adobe Acrobat Reader open the document without issues.  I do not suspect the document has been corrupted after its creation.
For reference the PDF was part of the ZIP archive linked on <http://repository.azgs.az.gov/uri_gin/azgs/dlio/1615 http://repository.azgs.az.gov/uri_gin/azgs/dlio/1615>.  It was created by someone at the Arizona Geologic Survey in 2015.
Hope this helps.
-brett

On Dec 12, 2019, at 8:40 PM, Ari Sosnovsky<EMAIL_ADDRESS>wrote:
Thanks for the kind words.
I like the idea of adding a counter to the number of wrong characters. I could implement an error-state and keep track of the number of bad-characters (as well as other errors that could propagate). Then i could raise a warning if anything has been logged to the error-state (plus make it accessible via PDFInterpreter.errors, where I could store some contextual information about the bad-characters, like where they were found).
So it would likely be a class like this
class ErrorState:
bad_chars: List[BadChar]
class BadChar(NamedTuple):
cid: int
lead_chars: List[str]
follow_chars: List[str]
page_num: int
What do you think? In the meanwhile I have implemented ignore_bad_chars flag, so that you can run this clearly, but it only replaces the bad-characters with "".
Some context, this library is actually just a refactoring project of the pdfminer.six project. Specifically, because I needed access to color and reliable rect positioning information (in the way I also took the liberty on modding the structure of repo, and added a bit of a speed-boost and more reliable html convertion). Given so, my understanding of CMap might not be any better than yours. The most I know is that it is a mapping of characters within the document. So if a cid is not found, that would mean that either;
I did not port the implementation well enough
the original writers in pdfminer made a mistake
the tool you used to make the pdf errored while writing the table and some of rows are missing (or perhaps the character itself was written wrong)
the pdf document was corrupted at one point
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/asosnovsky/pdfmajor/issues/2?email_source=notifications&email_token=ANKCUCPVXECKIMOWD5YOZQDQYLYZPA5CNFSM4JW3NT62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGYW23I#issuecomment-565276013, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANKCUCPXQBR22UE5HARZZWLQYLYZPANCNFSM4JW3NT6Q.

In the PDFInterpreter constructor there is now an option "ignore_bad_chars".  This is a good workaround.


Example deploy command failing

The default kpm deploy as demonstrated in the README is failing for me, complaining about a malformed variable.
[openshift@test-cluster-master-0 stackanetes]$ kpm deploy stackanetes/stackanetes --namespace openstack --variables stackanetes/parameters.yaml -H https://beta.kpm.sh
Traceback (most recent call last):
  File "/usr/bin/kpm", line 8, in <module>
    args.func(args)
  File "/usr/lib/python2.7/site-packages/kpm/command.py", line 40, in install
    variables = parse_cmdline_variables(options.variables)
  File "/usr/lib/python2.7/site-packages/kpm/utils.py", line 18, in parse_cmdline_variables
    raise ValueError("Malformed variable: %s" % v)
ValueError: Malformed variable: stackanetes/parameters.yaml

This happened with the stackanetes/parameters.yaml file as provided in the current head.
[openshift@test-cluster-master-0 stackanetes]$ git rev-parse HEAD
859c94ff25495bac0d4386c871a4e0b82b34ea9a

The yaml file is valid according to yamllint
Also, for what it's worth the kpm install command as shown in the readme didn't work for me either. I had to install using this method as described in the kpm docs:
[root@test-cluster-master-0 kpm]# sudo pip install kpm -U

Additionally, if I remove the --variables stackanetes/parameters.yaml I'm left with another error for a bad gateway for url beta.kpm.sh however I can reach that domain generally over the network with a ping.
[openshift@test-cluster-master-0 stackanetes]$ kpm deploy stackanetes/stackanetes --namespace openstack  -H https://beta.kpm.sh

Traceback (most recent call last):
  File "/usr/bin/kpm", line 8, in <module>
    args.func(args)
  File "/usr/lib/python2.7/site-packages/kpm/command.py", line 49, in install
    variables=variables)
  File "/usr/lib/python2.7/site-packages/kpm/deploy.py", line 83, in deploy
    return _process(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kpm/deploy.py", line 31, in _process
    packages = registry.generate(package_name, namespace=namespace, version=version, variables=variables)
  File "/usr/lib/python2.7/site-packages/kpm/registry.py", line 74, in generate
    r.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 893, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://beta.kpm.sh/api/v1/packages/stackanetes/stackanetes/generate?namespace=openstack
[openshift@test-cluster-master-0 stackanetes]$
[openshift@test-cluster-master-0 stackanetes]$ ping -c 2 beta.kpm.sh
PING beta.kpm.sh (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from 195-154-150-206.rev.poneytelecom.eu (<IP_ADDRESS>): icmp_seq=1 ttl=55 time=80.7 ms
64 bytes from 195-154-150-206.rev.poneytelecom.eu (<IP_ADDRESS>): icmp_seq=2 ttl=55 time=80.9 ms

Hey, which version of kpm are you using ?

Hey,
Unfortunately code in the public KPM registry is out of date. You'll need to setup your own registry:
kpm deploy coreos/kpm-registry --namespace kpm -H https://beta.kpm.sh
cd stackanetes
kpm push -H http://my-registry-address -f
and than you can deploy stackanetes with:
kpm deploy coreos/kpm-registry --namespace kpm -H http://my-registry-address
Let us know if that worked for you.
Cheers

Thanks ss7, I'll give it a try with my own kpm registry and let you know.
Also, just for what it's worth here's my kpm version.
[openshift@test-cluster-master-0 ~]$ pip show kpm | grep Version
Metadata-Version: 1.1
Version: 0.16.1

Hmm 0.16.1 is very old, that explains error you see :P Unfortunatley you need something much more recent, ie: https://pypi.python.org/pypi/kpm/0.21.1rc2
Cheers

Thanks again for the heads up on the version!
sudo yum install -y gcc-c++ python-devel

Somehow pip kept wanting to download 0.16.1 so I downloaded the tarball and ran
sudo pip install kpm.tar.gz

Which has me winding up with:
[openshift@test-cluster-master-0 ~]$ pip show kpm | grep Version
Version: 0.21.1rc2

Indeed -- that does keep it from complaining about the --variables ... parameter.
I'm going to try with my own kpm registry here shortly. Thanks again, appreciate the input a lot.

Alright, a fair bit closer, I got kpm registry up and running locally, and I can successfully push it, but, I go to deploy and I'm getting:
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

Here's my process generally:
[openshift@test-cluster-master-0 stackanetes]$ pwd
/home/openshift/stackanetes/stackanetes
[openshift@test-cluster-master-0 stackanetes]$ kpm push -H http://kpm-registry.kpm.svc.cluster.local -f
package: stackanetes/stackanetes (0.1.1) pushed
[openshift@test-cluster-master-0 stackanetes]$ cd ../
[openshift@test-cluster-master-0 stackanetes]$ kpm deploy stackanetes/stackanetes --namespace openstack-test --variables stackanetes/parameters.yaml -H http://kpm-registry.kpm.svc.cluster.local
Traceback (most recent call last):
  File "/usr/bin/kpm", line 6, in <module>
    cli()
  File "/usr/lib/python2.7/site-packages/kpm/command.py", line 63, in cli
    args.func(args)
  File "/usr/lib/python2.7/site-packages/kpm/commands/command_base.py", line 16, in call
    self(options)()
  File "/usr/lib/python2.7/site-packages/kpm/commands/command_base.py", line 19, in __call__
    self._call()
  File "/usr/lib/python2.7/site-packages/kpm/commands/deploy.py", line 60, in _call
    fmt=self.output)
  File "/usr/lib/python2.7/site-packages/kpm/deploy.py", line 89, in deploy
    return _process(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kpm/deploy.py", line 36, in _process
    shards=shards)
  File "/usr/lib/python2.7/site-packages/kpm/registry.py", line 83, in generate
    r = requests.get(self._url(path), data=json.dumps(body), params=params, headers=self.headers)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 473, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))
[openshift@test-cluster-master-0 stackanetes]$
[openshift@test-cluster-master-0 stackanetes]$ pip show kpm | grep Version
Version: 0.21.1rc2

Thanks again for taking a look.

@dougbtv Can you tell if the following patch is present on your kpm client? Anything in your kpm registry logs?
cc @ant31

@Quentin-M thanks for taking a look, I appreciate it. It appears that patch is present, I see the added line that wasn't there previously, by checking:
[root@test-cluster-master-0 stackanetes]# grep -n "gc_min_objects=9999999, gc_growth_trigger=9999999" /usr/lib/python2.7/site-packages/kpm/render_jsonnet.py
84:                                                 gc_min_objects=9999999, gc_growth_trigger=9999999)
[root@test-cluster-master-0 stackanetes]# echo $?
0

Additionally here's what I've got in my kpm logs (thanks for pointing me there)... this error is apparent:
HTTPError: 404 Client Error: NOT FOUND for url: http://kpm-registry.kpm.svc.cluster.local/api/v1/packages/stackanetes/mariadb/pull

I deploy the same as above, and get the same error when using the client. And then for logs in the registry (a snippet around the same time, sorry for the flood here)
INFO in info [/opt/kpm-0.21.1/kpm/api/info.py:25]:
request
--------------------------------------------------------------------------------
{"name": "kpm.api.app", "remote_addr": "<IP_ADDRESS>", "@timestamp": "2016-12-16T01:59:59.758574", "original_url": "http://<IP_ADDRESS>:5000/version", "funcName": "pre_request_logging", "headers": {"Connection": "close", "Host": "<IP_ADDRESS>:5000", "Accept-Encoding": "gzip", "User-Agent": "Go-http-client/1.1"}, "source_host": "kpm-registry-673535305-s5caf", "http_method": "GET", "lineno": 25, "path": "/version", "message": "request", "data": {}, "levelname": "INFO"}
--------------------------------------------------------------------------------
INFO in builder [/opt/kpm-0.21.1/kpm/api/builder.py:39]:
generate stackanetes/stackanetes
--------------------------------------------------------------------------------
{"lineno": 39, "source_host": "kpm-registry-673535305-s5caf", "name": "kpm.api.app", "@timestamp": "2016-12-16T02:00:01.247187", "message": "generate stackanetes/stackanetes", "levelname": "INFO", "funcName": "build"}
--------------------------------------------------------------------------------
INFO in registry [/opt/kpm-0.21.1/kpm/api/registry.py:47]:
pull stackanetes/stackanetes
--------------------------------------------------------------------------------
{"lineno": 47, "source_host": "kpm-registry-673535305-s5caf", "name": "kpm.api.app", "@timestamp": "2016-12-16T02:00:01.253617", "message": "pull stackanetes/stackanetes", "levelname": "INFO", "funcName": "pull"}
--------------------------------------------------------------------------------
INFO in registry [/opt/kpm-0.21.1/kpm/api/registry.py:47]:
pull stackanetes/mariadb
--------------------------------------------------------------------------------
{"lineno": 47, "source_host": "kpm-registry-673535305-s5caf", "name": "kpm.api.app", "@timestamp": "2016-12-16T02:00:03.920798", "message": "pull stackanetes/mariadb", "levelname": "INFO", "funcName": "pull"}
[2016-12-16 02:00:03 +0000] [11] [ERROR] Socket error processing request.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/usr/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 191, in handle_request
    six.reraise(*sys.exc_info())
  File "/usr/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 2000, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1991, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/usr/lib/python2.7/site-packages/flask_cors/extension.py", line 188, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1567, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python2.7/site-packages/flask_cors/extension.py", line 188, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/kpm-0.21.1/kpm/api/builder.py", line 41, in build
    return jsonify(k.build())
  File "/opt/kpm-0.21.1/kpm/kub_jsonnet.py", line 77, in build
    for kub in self.dependencies:
  File "/opt/kpm-0.21.1/kpm/kub_jsonnet.py", line 46, in dependencies
    self._fetch_deps()
  File "/opt/kpm-0.21.1/kpm/kub_base.py", line 128, in _fetch_deps
    namespace=self.namespace)
  File "/opt/kpm-0.21.1/kpm/kub_jsonnet.py", line 26, in __init__
    super(KubJsonnet, self).__init__(*args, **kwargs)
  File "/opt/kpm-0.21.1/kpm/kub_base.py", line 42, in __init__
    result = self._fetch_package()
  File "/opt/kpm-0.21.1/kpm/kub_base.py", line 105, in _fetch_package
    return self._registry.pull(self._deploy_name, self._deploy_version)
  File "/opt/kpm-0.21.1/kpm/registry.py", line 53, in pull
    r.raise_for_status()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 844, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 404 Client Error: NOT FOUND for url: http://kpm-registry.kpm.svc.cluster.local/api/v1/packages/stackanetes/mariadb/pull
--------------------------------------------------------------------------------
INFO in info [/opt/kpm-0.21.1/kpm/api/info.py:25]:
request
--------------------------------------------------------------------------------
{"name": "kpm.api.app", "remote_addr": "<IP_ADDRESS>", "@timestamp": "2016-12-16T02:00:09.758579", "original_url": "http://<IP_ADDRESS>:5000/version", "funcName": "pre_request_logging", "headers": {"Connection": "close", "Host": "<IP_ADDRESS>:5000", "Accept-Encoding": "gzip", "User-Agent": "Go-http-client/1.1"}, "source_host": "kpm-registry-673535305-s5caf", "http_method": "GET", "lineno": 25, "path": "/version", "message": "request", "data": {}, "levelname": "INFO"}
--------------------------------------------------------------------------------
INFO in info [/opt/kpm-0.21.1/kpm/api/info.py:25]:
request
--------------------------------------------------------------------------------
{"name": "kpm.api.app", "remote_addr": "<IP_ADDRESS>", "@timestamp": "2016-12-16T02:00:19.758946", "original_url": "http://<IP_ADDRESS>:5000/version", "funcName": "pre_request_logging", "headers": {"Connection": "close", "Host": "<IP_ADDRESS>:5000", "Accept-Encoding": "gzip", "User-Agent": "Go-http-client/1.1"}, "source_host": "kpm-registry-673535305-s5caf", "http_method": "GET", "lineno": 25, "path": "/version", "message": "request", "data": {}, "levelname": "INFO"}
--------------------------------------------------------------------------------
INFO in info [/opt/kpm-0.21.1/kpm/api/info.py:25]:

Hello,
First you have to push the "packages" from the repo to your local kpm-registry.
git clone https://github.com/stackanetes/stackanetes.git
cd stackanetes

# edit stackanetes/parameters.yaml ...

find -maxdepth 1  -type d -exec bash -c "cd {} ; [[ -f manifest.jsonnet ]] && kpm push -H http://kpm-registry.kpm -f" \;

Guys, for me this problem looks like kpm is not really working on CentOS. @dougbtv which CentOS version are you using ?
Cheers

@PAStheLoD thank you, that is helpful -- appreciate you brewing up the command for me. And hopefully when I get this ironed out I can help contribute to the docs to get that in there, too.
After pushing all the packages to the repo as suggested, when performing the kpm deploy I no longer see the 404 error in the kpm logs. Notably -- the deploy now takes a long time to fail (a few minutes). I don't see any obvious errors in the kpm logs, now.
e.g. via:
$ find -maxdepth 1  -type d -exec bash -c "cd {} ; [[ -f manifest.jsonnet ]] && kpm push -H http://kpm-registry.kpm.svc.cluster.local -f" \;
package: stackanetes/cinder (0.1.0) pushed
package: stackanetes/elasticsearch (0.1.0) pushed
package: stackanetes/etcd (3.0.6-1) pushed
package: stackanetes/glance (0.1.0) pushed
package: stackanetes/horizon (0.1.0) pushed
package: stackanetes/keystone (0.2.0) pushed
package: stackanetes/mariadb (0.3.0) pushed
package: stackanetes/memcached (0.1.0) pushed
package: stackanetes/neutron (0.2.0) pushed
package: stackanetes/nova (0.1.0) pushed
package: stackanetes/rabbitmq (0.2.0) pushed
package: stackanetes/searchlight (0.1.0) pushed
package: stackanetes/stackanetes (0.1.1) pushed
package: stackanetes/traefik (0.1.0) pushed

$ kpm deploy stackanetes/stackanetes --namespace openstack --variables stackanetes/parameters.yaml -H http://kpm-registry.kpm.svc.cluster.local
Traceback (most recent call last):
  File "/usr/bin/kpm", line 6, in <module>
    cli()
  File "/usr/lib/python2.7/site-packages/kpm/command.py", line 63, in cli
    args.func(args)
  File "/usr/lib/python2.7/site-packages/kpm/commands/command_base.py", line 16, in call
    self(options)()
  File "/usr/lib/python2.7/site-packages/kpm/commands/command_base.py", line 19, in __call__
    self._call()
  File "/usr/lib/python2.7/site-packages/kpm/commands/deploy.py", line 60, in _call
    fmt=self.output)
  File "/usr/lib/python2.7/site-packages/kpm/deploy.py", line 89, in deploy
    return _process(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kpm/deploy.py", line 36, in _process
    shards=shards)
  File "/usr/lib/python2.7/site-packages/kpm/registry.py", line 83, in generate
    r = requests.get(self._url(path), data=json.dumps(body), params=params, headers=self.headers)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 473, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

Here's my kpm registry logs http://pasteall.org/155023
@ss7pro I am on CentOS 7.2.1511 on this machine. CentOS 7.3 just was release a couple days ago (I think) for what it's worth.
$ uname -a
Linux test-cluster-master-0.localdomain 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

Try building kpm v0.21.2 from the repo.
With that the deploy succeeded for me. (I mean exited without error, the deployments got created on the kube cluster, then day turned to night instantly and the horrors came, oh the horrors. So, for some reason a lot of deployed things just failed with Error, and no termination log or any indication of why the pod is failed.)

Hi,
I think it's the server-side generation that take too much time or resources.
In later versions the generation is client-side and shows immediatly some outputs.
I'm going to upgrade the default client-version and  the api on beta.kpm.sh.
Also the connection seems aborted, can you check the gunicorn --timeout in the deployment command?

Ok, folks :) We figured out what's wrong :P It looks like etcd deployment for kpm-registry is brroken, etcd service aggregating three instances is not being created. @dougbtv Can you give me output of kubectl get svc ?

Just an fyi that after updating to v0.21.2 I'm having way better luck, installed like:
$ git clone https://github.com/coreos/kpm.git
$ cd kpm/
$ git checkout v0.21.2
$ sudo yum install -y npm
$ cd kpm-ui/
$ npm install
$ cd ../
$ sudo make install
$ kpm version
Api-version: .. Connection error
Client-version: 0.21.2

And now deploy is looking like:
[openshift@test-cluster-master-0 stackanetes]$ kpm deploy stackanetes/stackanetes --namespace openstack-test --variables stackanetes/parameters.yaml -H http://kpm-registry.kpm.svc.cluster.local
create stackanetes/stackanetes

 01 - stackanetes/mariadb:
 --> openstack-test (namespace): created
 --> mariadb-seedsh (configmap): created
 --> bootstrap-db (configmap): created
 --> mariadb-startsh (configmap): created
 --> mariadb-peer-finder (configmap): created
 --> mariadb-charsets (configmap): created
 --> mariadb-engine (configmap): created
 --> mariadb-mycnf (configmap): created
 --> mariadb-log (configmap): created
 --> mariadb-networking (configmap): created
[...snip...]

(I haven't let it finish yet)
@ss7pro nice finding! so.... here's what kubectl get svc looks like in the kpm namespace (e.g. for the registry...)
$ kubectl --namespace=kpm get svc
NAME           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
etcd           <IP_ADDRESS>   <none>        2379/TCP,4001/TCP,2380/TCP   1h
etcd-kpm-1     <IP_ADDRESS>   <none>        2379/TCP,4001/TCP,2380/TCP   1h
etcd-kpm-2     <IP_ADDRESS>   <none>        2379/TCP,4001/TCP,2380/TCP   1h
etcd-kpm-3     <IP_ADDRESS>    <none>        2379/TCP,4001/TCP,2380/TCP   1h
kpm-registry   <IP_ADDRESS>    <nodes>       80/TCP                       1h

@dougbtv So now everything is fine, deployment should succeed !

It looks like the kpm deployment succeeded, excellent! So that's great news. There's a few pods that are in error, but, I'm going to guess it's a configuration issue from not having done much change to stackanetes/parameters.yaml. So I wanna check that out and see if I can have the pods running.
Awesomeness, thank you for the help on the deploY!!

@dougbtv can you do kubectl logs pod-name-in-error and show us output here ?

Looks like it's mariadb-init that's failing with:
$ kubectl logs mariadb-init-1vcm3
Entrypoint INFO: 2016/12/16 16:03:50 entrypoint.go:47: Resolving /etc/my.cnf.d/wsrep.cnf
Entrypoint INFO: 2016/12/16 16:03:50 config.go:67: /etc/my.cnf.d/wsrep.cnf
Entrypoint INFO: 2016/12/16 16:03:52 entrypoint.go:56: Dependency /etc/my.cnf.d/wsrep.cnf is resolved
+ SLEEP_TIMEOUT=5
++ python /tmp/replicas.py
Traceback (most recent call last):
  File "/tmp/replicas.py", line 40, in <module>
    main()
  File "/tmp/replicas.py", line 33, in main
    reply = get_daemonsets()
  File "/tmp/replicas.py", line 27, in get_daemonsets
    response = urllib2.urlopen(req, context=ctx)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden
+ REPLICAS=

@dougbtv This is really bad, it means that https://github.com/stackanetes/stackanetes/blob/master/mariadb/templates/configmaps/replicas.py.yaml.j2 is unable to access kubernets API. It tries to query:
URL = ('https://kubernetes.default.svc.{{ network.dns.kubernetes_domain }}/apis/extensions/v1beta1/daemonsets')

to figure out desired number of galera instances. To access kubernetes api it uses:
TOKEN_FILE = '/var/run/secrets/kubernetes.io/serviceaccount/token'
It looks like it's unable to get list of all daemonSet running :(( Which version of kubernetes are you running ?

@dougbtv I can confirm that i can reprocude this problem on k8s 1.5 :(

Awww, bummer on it still on k8s 1.5 too. I'm running k8s 1.3.0.
Specifically...
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0+52492b4", GitCommit:"52492b4", GitTreeState:"clean", BuildDate:"2016-10-19T16:03:06Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0+52492b4", GitCommit:"52492b4", GitTreeState:"clean", BuildDate:"2016-10-19T16:03:06Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

I also entered the container and saw the url at the top of the python script there and tried to curl it, again with a 403 error.

@dougbtv  Can you execute them from the container:
KUBE_TOKEN=$(</var/run/secrets/kubernetes.io/serviceaccount/token)
curl -sSk -H "Authorization: Bearer $KUBE_TOKEN" https://kubernetes.default.svc.cluster.local/apis/extensions/v1beta1/daemonsets -k --verbose

Check this again and on my 1.5 it works, and confirmed that it works on my 1.3 cluster as well.

Here's what I've got looking at that...
[mysql@mariadb-init-t2215-debug /tmp]$ KUBE_TOKEN=$(</var/run/secrets/kubernet
es.io/serviceaccount/token)
curl -sSk -H "Authorization: Bearer $KUBE_TOKEN" https://kubernetes.default.svc.
cluster.local/apis/extensions/v1beta1/daemonsets -k --verbose()[mysql@mariadb-in
it-t2215-debug /tmp]$
* Hostname was NOT found in DNS cache
*   Trying <IP_ADDRESS>...
* Connected to kubernetes.default.svc.cluster.local (<IP_ADDRESS>) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*        subject: CN=<IP_ADDRESS>
*        start date: 2016-11-29 17:26:34 GMT
*        expire date: 2018-11-29 17:26:35 GMT
*        issuer: CN=openshift-signer@1480440393
*        SSL certificate verify result: self signed certificate in certificate c
hain (19), continuing anyway.
> GET /apis/extensions/v1beta1/daemonsets HTTP/1.1
> User-Agent: curl/7.38.0
> Host: kubernetes.default.svc.cluster.local
> Accept: */*
> Authorization: Bearer
>
< HTTP/1.1 403 Forbidden
< Cache-Control: no-store
< Content-Type: application/json
< Date: Fri, 16 Dec 2016 16:37:57 GMT
< Content-Length: 293
<
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "User \"system:anonymous\" cannot list all extensions.daemonsets in
 the cluster",
  "reason": "Forbidden",
  "details": {
    "group": "extensions",
    "kind": "daemonsets"
  },
  "code": 403
* Connection #0 to host kubernetes.default.svc.cluster.local left intact
()[mysql@mariadb-init-t2215-debug /tmp]$

Ahhhh, maybe I need to grant some access to system:anonymous, huh? I'm going to look into that.

@dougbtv Yes, we operate on a default settings for authorization in kubernetes, where serviceaccount token can interact with API.

Awesome, I'm about to enter a meeting and eat some lunch. I'll come back to this a little later, but, that's a great direction for me to look in. Also -- just wanted to make sure i say "thank you!!!" a ton for helping me out here, I appreciate it greatly.

I wound up removing the pods and deploying again. I'm having trouble on subsequent deploys somehow, I'm getting:
subprocess.CalledProcessError: Command '['kubectl', 'create', '-f', '/tmp/tmpmd6pOi', '--namespace', 'openstack-again']' returned non-zero exit status 1

Oddly enough on multiple attempts it will fail on different deployments (this time on keystone, but previously on glance [twice] and nova).
Here's my entire log:
[openshift@test-cluster-master-0 stackanetes]$ kpm deploy stackanetes/stackanetes --namespace openstack-again --variables stackanetes/parameters.yaml -H http://kpm-registry.kpm.svc.cluster.local
create stackanetes/stackanetes

 01 - stackanetes/mariadb:
 --> openstack-again (namespace): created
 --> mariadb-seedsh (configmap): created
 --> bootstrap-db (configmap): created
 --> mariadb-startsh (configmap): created
 --> mariadb-peer-finder (configmap): created
 --> mariadb-charsets (configmap): created
 --> mariadb-engine (configmap): created
 --> mariadb-mycnf (configmap): created
 --> mariadb-log (configmap): created
 --> mariadb-networking (configmap): created
 --> mariadb-pid (configmap): created
 --> mariadb-tuning (configmap): created
 --> mariadb-replicas (configmap): created
 --> mariadb-wsrep (configmap): created
 --> mariadb-readiness (configmap): created
 --> mariadb-init (job): created
 --> mariadb (daemonset): created
 --> mariadb (service): created

 02 - stackanetes/rabbitmq:
 --> openstack-again (namespace): ok
 --> rabbitmq-enabledplugins (configmap): created
 --> rabbitmq-rabbitmqenvconf (configmap): created
 --> rabbitmq-rabbitmqconfig (configmap): created
 --> rabbitmq-rabbitmqdefinitions (configmap): created
 --> rabbitmq-scripts (configmap): created
 --> rabbitmq-erlangcookie (configmap): created
 --> rabbitmq (daemonset): created
 --> rabbitmq (service): created

 03 - stackanetes/memcached:
 --> openstack-again (namespace): ok
 --> memcached (deployment): created
 --> memcached (service): created

 04 - stackanetes/etcd:
 --> openstack-again (namespace): ok
 --> etcd-1 (deployment): created
 --> etcd-2 (deployment): created
 --> etcd-3 (deployment): created
 --> etcd-1 (service): created
 --> etcd-2 (service): created
 --> etcd-3 (service): created
 --> etcd (service): created

 05 - stackanetes/elasticsearch:
 --> openstack-again (namespace): ok
 --> searchlight-es (deployment): created
 --> searchlight-es (service): created

 06 - stackanetes/keystone:
 --> openstack-again (namespace): ok
 --> keystone-initsh (configmap): created
 --> keystone-dbsyncsh (configmap): created
 --> keystone-startsh (configmap): created
 --> keystone-keystoneconf (configmap): created
 --> keystone-wsgikeystone (configmap): created
 --> keystone-mpmeventconf (configmap): created
 --> keystone-init (job): created
 --> keystone-db-sync (job): created
 --> keystone-api (deployment): created
Traceback (most recent call last):
  File "/usr/bin/kpm", line 4, in <module>
    __import__('pkg_resources').run_script('kpm==0.21.2rc0', 'kpm')
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 739, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1494, in run_script
    exec(code, namespace, namespace)
  File "/usr/lib/python2.7/site-packages/kpm-0.21.2rc0-py2.7.egg/EGG-INFO/scripts/kpm", line 6, in <module>
    cli()
  File "/usr/lib/python2.7/site-packages/kpm-0.21.2rc0-py2.7.egg/kpm/command.py", line 64, in cli
    args.func(args)
  File "/usr/lib/python2.7/site-packages/kpm-0.21.2rc0-py2.7.egg/kpm/commands/command_base.py", line 18, in call
    self(options)()
  File "/usr/lib/python2.7/site-packages/kpm-0.21.2rc0-py2.7.egg/kpm/commands/command_base.py", line 21, in __call__
    self._call()
  File "/usr/lib/python2.7/site-packages/kpm-0.21.2rc0-py2.7.egg/kpm/commands/deploy.py", line 74, in _call
    fmt=self.output)
  File "/usr/lib/python2.7/site-packages/kpm-0.21.2rc0-py2.7.egg/kpm/formats/kub.py", line 182, in deploy
    return self._process_deploy(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kpm-0.21.2rc0-py2.7.egg/kpm/formats/kub.py", line 156, in _process_deploy
    status = getattr(kubresource, action)(force=force, dry=dry)
  File "/usr/lib/python2.7/site-packages/kpm-0.21.2rc0-py2.7.egg/kpm/platforms/kubernetes.py", line 113, in create
    self._call(cmd, dry=dry)
  File "/usr/lib/python2.7/site-packages/kpm-0.21.2rc0-py2.7.egg/kpm/platforms/kubernetes.py", line 177, in _call
    return subprocess.check_output(command, stderr=subprocess.STDOUT)
  File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['kubectl', 'create', '-f', '/tmp/tmpmd6pOi', '--namespace', 'openstack-again']' returned non-zero exit status 1
[openshift@test-cluster-master-0 stackanetes]$

@dougbtv in /tmp/stackanetes you should see all expanded kubernetes  manifest.
You can try to kubectl create -f the one that failed

Thanks ant31 -- in my case doing a rm -Rf /tmp/stackanetes before redeploying works wonders and prevents the error from re-occurring.
I was able to fix up the k8s api access to allow the service account to properly access, and a lot more seems to be chugging along.
Currently, some pods are failing with complaints that look like they can't access mariadb, and I don't see any other maria db pods running
Here's the logs from the completed mariadb-init pod.
[root@testkpm templates]# kubectl logs mariadb-init-7coec
Entrypoint INFO: 2016/12/19 00:32:28 entrypoint.go:47: Resolving /etc/my.cnf.d/wsrep.cnf
Entrypoint INFO: 2016/12/19 00:32:28 config.go:67: /etc/my.cnf.d/wsrep.cnf
Entrypoint INFO: 2016/12/19 00:32:30 entrypoint.go:56: Dependency /etc/my.cnf.d/wsrep.cnf is resolved
+ SLEEP_TIMEOUT=5
++ python /tmp/replicas.py
+ REPLICAS=1
+ '[' 1 -eq 1 ']'
+ echo 'Requested to build one-instance MariaDB cluster. There is no need to run seed. Exiting.'
+ exit 0
Requested to build one-instance MariaDB cluster. There is no need to run seed. Exiting.
[root@testkpm templates]#
[root@testkpm templates]#
[root@testkpm templates]# kubectl get pods --show-all
NAME                                READY     STATUS             RESTARTS   AGE
etcd-1-2020413439-8xy34             1/1       Running            1          14m
etcd-2-2940211204-my1qc             1/1       Running            0          14m
etcd-3-3860008969-o2kip             1/1       Running            1          14m
glance-api-3697931357-pydw5         0/1       Running            0          13m
glance-db-sync-37970                1/1       Running            0          13m
glance-init-p35m8                   0/1       CrashLoopBackOff   6          13m
glance-post-bu99z                   0/1       CrashLoopBackOff   4          13m
glance-registry-4079849578-ul179    0/1       Running            0          13m
horizon-623622820-wttcy             1/1       Running            0          10m
keystone-api-1088402996-9j1rd       0/1       Running            0          14m
keystone-db-sync-o5d1v              1/1       Running            0          14m
keystone-init-asgh2                 0/1       CrashLoopBackOff   6          14m
mariadb-init-7coec                  0/1       Completed          0          15m
memcached-2936627269-1yg43          1/1       Running            0          14m
neutron-db-sync-a18fe               1/1       Running            0          10m
neutron-init-f23fo                  0/1       CrashLoopBackOff   6          10m
neutron-post-6sbk6                  1/1       Running            0          10m
neutron-server-3034663805-3nkff     0/1       Running            0          10m
nova-api-286364150-6axid            0/1       Running            0          12m
nova-conductor-3873691898-thnd4     1/1       Running            0          12m
nova-consoleauth-3382303763-ol2zb   1/1       Running            0          12m
nova-db-sync-lekdk                  1/1       Running            0          12m
nova-init-yjt2p                     0/1       CrashLoopBackOff   6          12m
nova-novncproxy-1061318108-skuvw    0/1       Running            0          11m
nova-post-yzaia                     1/1       Running            5          12m
nova-scheduler-616814770-1w5w1      1/1       Running            0          12m
rabbitmq-v46az                      1/1       Running            0          14m
searchlight-api-1228977481-er6yt    0/1       Running            0          10m
searchlight-es-287665420-qz8yh      1/1       Running            0          14m
searchlight-init-jkawg              0/1       CrashLoopBackOff   3          10m
traefik-3994158533-cnkl4            1/1       Running            0          9m

@dougbtv You need to enable hostPath volumes, https://docs.openshift.org/latest/admin_guide/manage_scc.html#use-the-hostpath-volume-plugin
You will also need adjust scc (security context constraints) to allow:
hostPid, hostNetwork, privileged, root in containers, allow USER from Dockerfile, all is documented here:
https://docs.openshift.org/latest/admin_guide/manage_scc.html#how-do-i
Sorry for all this, but we never validated our solution against openshift :P

Thanks for the pointers, and links to the docs, I appreciate it!
Also -- no need for apologies, I'm hoping to come out the other end with some documentation to share on how to get it all spun up on OpenShift, taking a lot of notes as I go along. Again, have appreciated the help here. Going to give it some more attention here hopefully later this morning or afternoon.

Alright, feeling like I'm fairly close at this point... I created a custom scc and mariadb (and a lot more) comes up very nicely. I've also taken to memorializing my steps so far in an ansible playbook that I have available here on github.
I'm stuck with 3 pods (neutron-openvswitch [with openvswitch-agent container failing], nova-libvirt, and nova-compute) all coming up with the same error:
Entrypoint Error: 2016/12/21 15:04:56 config.go:51: Cannot get iface: route ip+net: no such network interface

Any ideas what might be causing that? I'm continuing to dig at it, thanks.

It looks like that minion_interface_name you provoded in parameters.yaml is not found on this node. Can you check that ?

HI,
this error came from here https://github.com/stackanetes/kubernetes-entrypoint/blob/master/util/util.go#L17. It means that kubernetes-entrypoint cannon find interface, which you provided in INTERFACE_NAME environment variable.

What value you have in here :
https://github.com/stackanetes/stackanetes/blob/master/stackanetes/parameters.yaml#L17 ?

@PiotrProkop and @ss7pro thank you so much!! Thank you guys!!! I set minion_interface_name to the primary interface on the (single) node that I have, and.... voila!!! Everything is up and running.... I'm very happy -- what an amazing accomplishment that you guys have OpenStack running on k8s like this, I'm impressed with your work.
I'm going to write up some documentation on stackanetes on openshift and I'll share it with you all.
I can't thank you all enough, I would not have been able to get it to run without the assistance -- really appreciated the help a lot, and the expert eyes.

Good  happy to hear that !

Thank you!

Just wanted to give you guys a quick update that I went ahead and formalized what I learned here in order to get Stackanetes working on OpenShift.
First off, I created this project, a method of spinning up Stackanetes on openshift using Ansible, openshift-stackanetes: https://github.com/dougbtv/openshift-stackanetes
And I also went and documented some details about how that works here at my blog: http://dougbtv.com/nfvpe/2017/01/09/stackanetes-on-openshift/
Happy new year all, and thanks again for the help, really appreciated it.

@dougbtv Thank you for the blog post, it is indeed valuable information / feedback! We are looking forward to improving support for OpenStack on Kubernetes / OpenShift and significantly help the community out there.

Thanks @QuentinM ! Feel free to reach out, I'm excited to see it run on
OpenShift, and happy to contribute/collaborate there where I can.
On Mon, Jan 9, 2017, 8:55 PM Quentin Machu<EMAIL_ADDRESS>wrote:

@dougbtv https://github.com/dougbtv Thank you for the blog post, it is
indeed valuable information / feedback! We are looking forward to improving
support for OpenStack on Kubernetes / OpenShift and significantly help the
community out there.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/stackanetes/stackanetes/issues/144#issuecomment-271464445,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFN_vQG7CXAQQoATsWh5A9jtn60-jr3Bks5rQuT3gaJpZM4LLCdW
.

Hello,
I am having the same error mentioned previously on mariadb:

File "/usr/lib/python2.7/urllib2.py", line 550, in http_response     'http', request, response, code, msg, hdrs)   File "/usr/lib/python2.7/urllib2.py", line 475, in error     return self._call_chain(*args)   File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain     result = func(*args)   File "/usr/lib/python2.7/urllib2.py", line 558, in http_error_default     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 403: Forbidden  REPLICAS=

Could you please share how you did manage to solve it?
I am trying to deploy over a Kubernetes cluster managed by Tectonic using all default configuration.
Thank you a lot.

@gkatsaros You simply have RBAC enabled and therefore needs to add some permissions:
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  name: stackanetes-deployment
subjects:
  - kind: ServiceAccount
    name: default
    namespace: <YOUR_DEPLOYMENT_NAMESPACE>
roleRef:
  kind: ClusterRole
  # name of an existing ClusterRole, either "readonly", "user", "admin",
  # or a custom defined role.
  name: admin
  apiGroup: rbac.authorization.k8s.io

Hi @Quentin-M .
Thx a lot for the help. It indeed solved my previous problem and I managed to get mariadb up and running.
All of the pods seem to be working fine apart from the nova-compute pod which crashes with CrashLoopBackOff status and at the logs I see:
Entrypoint WARNING: 2017/03/01 17:56:44 entrypoint.go:52: Resolving dependency for neutron-openvswitch failed: daemonsets.extensions "neutron-openvswitch" not found Entrypoint WARNING: 2017/03/01 17:56:44 entrypoint.go:52: Resolving dependency for nova-post failed: Job nova-post is not completed yet
My cluster is a 2 minion CoreOS nodes with Kubernetes managed through Tectonic.
Any ideas?
Thank you very much in advance.

@gkatsaros I'd look at the logs for any containers for the nova-post initialization job. It must complete.

@Quentin-M
The nova-post job seems to complete.
The nova compute container shows the following error:
'Events:
FirstSeen     LastSeen        Count   From                                            SubObjectPath                   Type            Reason                  Message

2h            2m              37      {kubelet node1.tectonic.dev.inmarsat.com}       spec.containers{nova-compute}   Normal          Pulling                 pulling image "quay.io/stackanetes/stackanetes-nova-compute:newton"
2h            2m              37      {kubelet node1.tectonic.dev.inmarsat.com}       spec.containers{nova-compute}   Normal          Pulled                  Successfully pulled image "quay.io/stackanetes/stackanetes-nova-compute:newton"
2h            2m              28      {kubelet node1.tectonic.dev.inmarsat.com}       spec.containers{nova-compute}   Normal          Created                 (events with common reason combined)
2h            2m              37      {kubelet node1.tectonic.dev.inmarsat.com}       spec.containers{nova-compute}   Warning         FailedPostStartHook
2h            2m              28      {kubelet node1.tectonic.dev.inmarsat.com}       spec.containers{nova-compute}   Normal          Started                 (events with common reason combined)
2h            2m              37      {kubelet node1.tectonic.dev.inmarsat.com}                                       Warning         FailedSync              Error syncing pod, skipping: failed to "StartContainer" for "nova-compute" with RunContainerError: "PostStart handler: Error executing in Docker Container: 1"
2h    2m      28      {kubelet node1.tectonic.dev.inmarsat.com}       spec.containers{nova-compute}   Normal  Killing         (events with common reason combined)
2h    10s     747     {kubelet node1.tectonic.dev.inmarsat.com}       spec.containers{nova-compute}   Warning BackOff         Back-off restarting failed docker container
2h    10s     716     {kubelet node1.tectonic.dev.inmarsat.com}                                       Warning FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "nova-compute" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=nova-compute pod=nova-compute-2591303302-mzql4_openstack(5bac2316-ff2f-11e6-924d-74a2e6d39a73)"
'
All the rest of jobs and pods seem to be fine.
Do you have any suggestion on how to debug it?
Thank you a lot, I really appreciate your help.

Hmm ... I installed Stackanetes on Tectonic no later than yesterday without seeing any issues.
nova-compute's PostStart script tries to talk to keystone-api/nova-api to enable its hypervisor. The script failing might suggest that the Keystone API or the Nova API are not available? Is there anything there?


Make sure to include tests for --no-dead-storage flag on

The issue with the repolinter test failing is unrelated to your work. That  issue will be fixed in https://github.com/hyperledger-labs/solang/pull/405
Thank you @mmagician


Enable support for no access tokens

Closes #2029

Coverage increased (+0.06%) to 78.493% when pulling c2d4eaeb3282a2ae484bac9e76adc2850c7e1a61 on eatonphil:allow-no-scope-tokens into c098eb10673e356358a5d00946ea1643a8053fd6 on linode:develop.

Tested, works

Thank you!


Installing Ember Bootstrap throws in a newly generated ember app with Ember CLI 3.20.0

I just noticed that ember install ember-bootstrap in a newly generated Ember application with Ember CLI 3.20.0 throws:
$ ember install ember-bootstrap

🚧  Installing packages... This might take a couple of minutes.
Yarn: Installed ember-bootstrap
_1.MacrosConfig.astPlugins is not a function

Stack Trace and Error Report: /tmp/error.dump.00e2b30443199e52ab958e6038db1fd8.log
An error occurred in the constructor for ember-bootstrap at /home/jelhan/tmp/test-ember-bootstrap-form-with-custom-control/node_modules/ember-bootstrap

ERROR Summary:

  - broccoliBuilderErrorStack: [undefined]
  - code: [undefined]
  - codeFrame: [undefined]
  - errorMessage: _1.MacrosConfig.astPlugins is not a function
  - errorType: [undefined]
  - location:
    - column: [undefined]
    - file: [undefined]
    - line: [undefined]
  - message: _1.MacrosConfig.astPlugins is not a function
  - name: TypeError
  - nodeAnnotation: [undefined]
  - nodeName: [undefined]
  - originalErrorMessage: [undefined]
  - stack: TypeError: _1.MacrosConfig.astPlugins is not a function
    at Class.setupPreprocessorRegistry (/home/jelhan/tmp/test-ember-bootstrap-form-with-custom-control/node_modules/@embroider/macros/src/ember-addon-main.js:73:58)
    at /home/jelhan/tmp/test-ember-bootstrap-form-with-custom-control/node_modules/ember-cli-preprocess-registry/preprocessors.js:27:13
    at Array.forEach (<anonymous>)
    at setupRegistryForEachAddon (/home/jelhan/tmp/test-ember-bootstrap-form-with-custom-control/node_modules/ember-cli-preprocess-registry/preprocessors.js:25:10)
    at Object.module.exports.setupRegistry (/home/jelhan/tmp/test-ember-bootstrap-form-with-custom-control/node_modules/ember-cli-preprocess-registry/preprocessors.js:46:3)
    at Class.init (/home/jelhan/tmp/test-ember-bootstrap-form-with-custom-control/node_modules/ember-cli/lib/models/addon.js:323:7)
    at Class.superWrapper [as init] (/home/jelhan/tmp/test-ember-bootstrap-form-with-custom-control/node_modules/core-object/lib/assign-properties.js:34:20)
    at Class.init (/home/jelhan/tmp/test-ember-bootstrap-form-with-custom-control/node_modules/ember-bootstrap/index.js:43:22)
    at Class.superWrapper [as init] (/home/jelhan/tmp/test-ember-bootstrap-form-with-custom-control/node_modules/core-object/lib/assign-properties.js:34:20)
    at new CoreObject (/home/jelhan/tmp/test-ember-bootstrap-form-with-custom-control/node_modules/core-object/core-object.js:9:15)

Steps to reproduce:

npx<EMAIL_ADDRESS>new test-project
cd test-project
ember install ember-bootstrap

The error seems to be related to @embroider/macros. Used version of @embroider/marcos is 0.21.0. It throws in this line: https://github.com/embroider-build/embroider/blob/6e8b52395ddade767cf36c9c3dc687dd92d6d531/packages/macros/src/ember-addon-main.ts#L76-L83 Please note that line numbers do not match with the error stack trace cause it's using TypeScript and the line numbers in the error stack trace refer to the compiled JavaScript.
All Ember Bootstrap does before is calling super on init hook: https://github.com/kaliber5/ember-bootstrap/blob/9fceefb1f8702eee2de6736f86e8a4ccac4f9681/index.js#L42-L43 So it might not be related to Ember Bootstrap at all but a general issue with `@embroider/macros.
It's working fine if installing Ember Bootstrap and running it's blueprints separetly: yarn add -D ember-bootstrap && ember generate ember-bootstrap instead of ember install ember-bootstrap.
Current Ember Bootstrap version when reporting is 4.0.1. I have not tested with any other combination than<EMAIL_ADDRESS>and<EMAIL_ADDRESS>

I saw the identical error with Ember CLI 3.18 and Ember Bootstrap 4.0.1 yesterday as well

Confirmed. That's pretty bad actually 😱
Will investigate, however this is difficult to debug, as it seems this only happens when installing indeed. So no chance to just put a console.log somewhere in @embroider/macros, as that does not exist yet in node_modules 😟

Will investigate, however this is difficult to debug, as it seems this only happens when installing indeed. So no chance to just put a console.log somewhere in @embroider/macros, as that does not exist yet in node_modules worried

It may work if you are using a monorepo setup. But haven't tested if the error could be reproduced in that setup.

That was a really tricky one. Reported my investigation here: https://github.com/embroider-build/embroider/issues/509
Seems we cannot really do something on this addon's side. Hope this gets resolved somehow.
In the meantime will keep this issue open, for visibility. The workaround for users running into this issue is as mentioned above to first install the addon with the native tools (yarn add -D ember-bootstrap or npm install -D ember-bootstrap), and then run the blueprint, either the one with default settings as ember generate ember-bootstrap, or any of the more advanced variants as explained in the setup guide.

The workaround for users running into this issue is as mentioned above to first install the addon with the native tools (yarn add -D ember-bootstrap or npm install -D ember-bootstrap), and then run the blueprint, either the one with default settings as ember generate ember-bootstrap, or any of the more advanced variants as explained in the setup guide.

Even easier: Run the install command twice. First run will fail but second run will pass (cause first run has modified the dependencies).

Closed by https://github.com/ef4/ember-auto-import/pull/308, can confirm this does not happen anymore with a fresh new app having the latest ember-auto-import.


org.apache.hudi.exception.HoodieIOException: IOException when reading log file

I have loaded a Merge On Read table via the Bulk Insert API.  This will be an entity based table 315M Entities (2TB Parquet) and partitioned evenly into 1000 distinct partitions 0 -> 999.  The Bulk API works as expected and I am able to query the table via Presto/Hive/Spark.  When I start a live ingestion into this table I am able to ingest about 8 batches before I receive a corrupt log file exception shown below.
Exception killing jobs
20/08/26 01:11:26 ERROR SplunkStreamListener: |exception=org.apache.spark.SparkException
org.apache.spark.SparkException: Job aborted due to stage failure: Task 496 in stage 45.0 failed 4 times, most recent failure: Lost task 496.3 in stage 45.0 (TID 46608, <IP_ADDRESS>, executor 10): org.apache.hudi.exception.HoodieIOException: IOException when reading log file
I have duplicated this twice with fresh table loads against the like stream from Kafka.  Once the job dies and restarts job continues to have this exception.
Note: In parallel I also was streaming data into another non-bootstraped endpoint and the job is still running after 24 hours (~9M Inserts)
googling I was able to see a closed ticket however the end user only said to enable hoodie.consistency.check.enabled which did not resolve the issue.
First Failure (Table C) 8 Deltacommits to failure
Image from Hudi CLI for the table

First Failure (Table D) 7 Deltacommits to failure

o failure
Streaming (No Bootstrap CLI (Table Prod) 24-hours no failure yet [~48 batches completed so far]

All these jobs are running with the same code paths, and storage containers, just writing to different endpoints with updated configuration to change Kafka consumer groups, path locations.
Hudi Configuration:
"hoodie.compaction.strategy" = "org.apache.hudi.table.action.compact.strategy.UnBoundedCompactionStrategy",
"hoodie.consistency.check.enabled" -> "true"
DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL
HoodieCompactionConfig.INLINE_COMPACT_PROP = true,
DataSourceWriteOptions.HIVE_SYNC_ENABLED_OPT_KEY = true,
HoodieWriteConfig.UPSERT_PARALLELISM = 512,
HoodieWriteConfig.INSERT_PARALLELISM = 512,
HoodieCompactionConfig.CLEANER_COMMITS_RETAINED_PROP = 1,
HoodieCompactionConfig.INLINE_COMPACT_NUM_DELTA_COMMITS_PROP = 8,
HoodieStorageConfig.PARQUET_FILE_MAX_BYTES = 256 * 1024 * 1024
HoodieStorageConfig.PARQUET_FILE_MAX_BYTES = 256 * 1024 * 1024
Versions
Hudi: Release 0.6.0
Spark: 2.4.5 on Databricks
Storage: Azure ADLS_V2
Configuration
Spark Cluster for Streaming ingestion: 32-Core [128GB Ram] x 16 Executors
Most Importantly:
How would I recover from this failure mode?
What could be causing this failure mode and how would I avoid it?
Other thoughts around performance tuning:
Table is about 2TB and I am attempting to ingest and compact about 100GB worth of data per hour into this table.  Any other hints tips for managing this size table with N-minute batches (currently 15) and in-line compaction.

@n3nash : Can you help take a look at this ?
@sam-wmt : Can you please provide the full stack track of the corrupted log file exception ?

Thanks @bvaradar I will provide shortly

This appears to have been caused by an internal change to our Hudi writer which I found in the executor logs:
java.lang.NoSuchMethodException: com.xxx.xxxx.xxxxx.xxxxxx.(org.apache.hudi.common.util.Option)
.
Closing ticket


test(dtslint): add dtslint tests for windowCount operator (#4093)

Description: add dtslint tests for the windowCount operator
Related issue (if exists): #4093

Pull Request Test Coverage Report for Build 7681

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.2%) to 96.806%

Totals

Change from base Build 7679:
-0.2%

Covered Lines:
5244

Relevant Lines:
5417

💛  - Coveralls


Add channel label to embeds

We should visually indicate what channel is currently being used in the embeds (set by the channel query parameter.)

Closing in favor of a consolidated information menu in the embed layout: https://github.com/dart-lang/dart-pad/issues/3007


Fixed GUI to only show name and index in Others panel

After friend-ing someone, the card will be revealed to show everything about the user.

To help with the approving of the code:
I've changed the implementation to let the friend list take "personListPanel" instead of the previous one which was "otherListPanel", as now in otherListPanel there are some arguments which are set to null, so it's easier to use an existing unchanged panel for the personListPanel tests instead.

If only there are better aliases that are not taken, hahaha thanks Ben


NTR for CC of PMID:35536002

NTR: abolished inositol diphosphatase activity == inositol phosphate phosphatase activity
NTR: normal inositol hexakisphosphate kinase activity

abolished inositol diphosphatase activity == inositol phosphate phosphatase activity
already existed FYPO:0005485


[Bug] Spell checker isn't working

Describe the bug
The tile is enough
To Reproduce
Enable spell checker from settings (eventually close and reopen the app) and start writing misspelled words
Expected behavior
Highlight misspelled words
Screenshots

Desktop (please complete the following information):

OS: [e.g. Windows 10 1809 17763.593] Windows 10 1903
Version [e.g. v<IP_ADDRESS>] <IP_ADDRESS>

Additional context
None

Did you enable windows internal spell checker?

Did you enable windows internal spell checker?

You are right, now it's working👍

@JasonStein why don't you add a small dialog explaining that it's necessary to enable system one to have it working or remove the setting and make it follow system's choice?

@JasonStein why don't you add a small dialog explaining that it's necessary to enable system one to have it working or remove the setting and make it follow system's choice?

There are too many settings that might overwrite Notepads behaviors including the background transparency and acrylic effect and there are more that I am not even aware of. ScrollViewer is another thing. At the end of the day, user should be aware of since they are in control of the system settings and Notepads works with default settings.

I don't know, but some UWPs follow system settings for spell checker

@JasonStein spell checker is totally broken now


VIP scores in PLS regression are too small

The bug is most probably caused by using wrong scaling factor (number of predictors instead of number of measurements).

fixed, will be available in 0.1.7.


Add Slightly (Smil|Frown)ing Face PNGs

In https://github.com/twitter/twemoji/commit/d0e45fee1b7d2b5828de9d4e5c01acb84ef4a116 and https://github.com/twitter/twemoji/commit/b3d9b74ced05b6ec93b73c7aa1214f057212450a the AI and SVG files for Slightly Frowning/Smiling Face emojis U+1F641/2 have been added to Twemoji v1, but there are still no PNGs of them in the repo.

v1 of Twemoji is deprecated. Pleaes referring to v2 for current assets.

I know that it's deprecated, but some people are still using it, maybe just for comparison, and it makes no sense that the SVG files are available but the PNG files are not.


Introduced yAxisWidth and related layout

To align with use and exposure of xAxisHeight

One small comment but otherwise :+1:

Thanks for the contribution, I'm definitely :+1: to the idea of more configuration. Although I am wondering what this chart looks like if the y-axis width is != 0?

@ColinEberhardt I agree long-term it's probably not the right fix, but the same holds for the xAxisHeight so I think being consistent short-term is an ok fix.

Far enough, if it fixes an immediate need, I'm happy :-)

Merged, thanks @djmiley

:+1:


Bytea postgreql helper should accept a slice of bytes

Currently it accepts a string which is a bit counter intuitive considering the model is also using []byte.
This leads to more verbose input code: Bytea(string(myHash[:]))).
Thanks!

Yeah, I agree. Unfortunately Bytea function is part of public API, so we can't update it until next major version is released.
For now, we could add a new function, for example ByteaB that will accept byte array.

Actually the current implementation just doesn't work. If I insert data in a Bytea column with a slice, I have no way to do a comparison on it.

Actually the current implementation just doesn't work. If I insert data in a Bytea column with a slice, I have no way to do a comparison on it.

@Sytten Could you share you example?

@Sytten Could you share you example?

Here is an example:

hash := sha256.Sum256([]byte("somekey"))

// Store
_, _ = ReportingKey.INSERT(ReportingKey.Hash, ReportingKey.Name).
	VALUES(hash[:], input.Name).
	RETURNING(ReportingKey.AllColumns).
	Exec(db)

// Compare
stmt := SELECT(
		ReportingKey.AllColumns,
	).FROM(
		ReportingKey,
	).WHERE(
		ReportingKey.Hash.EQ(Bytea(string(myHash[:]))),
	)

// Query
var dest []model.ReportingKey
_ = stmt.Query(s.db, &dest)

// dest is empty here

I also try the hex.EncodeToString without any success because it misses the \x in the written string.

Here is an example:

hash := sha256.Sum256([]byte("somekey"))

// Store
_, _ = ReportingKey.INSERT(ReportingKey.Hash, ReportingKey.Name).
	VALUES(hash[:], input.Name).
	RETURNING(ReportingKey.AllColumns).
	Exec(db)

// Compare
stmt := SELECT(
		ReportingKey.AllColumns,
	).FROM(
		ReportingKey,
	).WHERE(
		ReportingKey.Hash.EQ(Bytea(string(myHash[:]))),
	)

// Query
var dest []model.ReportingKey
_ = stmt.Query(s.db, &dest)

// dest is empty here

I also try the hex.EncodeToString without any success because it misses the \x in the written string.

Aha, I see.

Aha, I see.

Bytea constructor now accepts byte array as well as string in the latests v2.5.0 release.


Laptop monitor errors

Great tool! Super handy, however when I'm working on my laptop Crystools inevitably will through errors around power and the monitor.
It would be nice if Crystools didn't throw errors when my laptop monitor turns off while a ComfyUI gen is happening. I shouldn't have to keep the monitor active during a long generation to avoid Crystools errors. I also shouldn't get errors if I'm running on battery vs being plugged in.
Is there a way to work around this?
Thank you for your help!

I need more info, but, is an issue from the driver of nvidia :(


How can a FS provider determine the correct workspace folder root when it recevies a call

Consider a FS provider that wants to show the content of a zip file. It therefore registers a provider for a schema zip. It also allows to open a zip by using a folder uri zip:///home/dbaeumer/myzip.zip.
When the FS provider receives the first call for an URI (which can look like zip:///home/dbaeumer/myzip.zip/.vscode) it needs to know which part of the URI is the root folder to open the zip. Searching for something like name.zip doesn't work since a folder could also be named myDir.zip. I see the following options:

we guaruntee that the first call is always a stat call with the root folder.
we guaruntee that the workspace.workspaceFolders is containing the root before the first call. We also guaruntee that:

if a folder gets added a workspace folder add event is sent before the first call to the FS provider.
if a folder gets removed it would be nice if no more calls to the FS provider are made. But an extensions could handle this gracefully knowing it has to.

we always pass in the root to the FS provider calls.
we add some handle & unhandle methods to the FS provider which are called if the provider should handle new roots or roots it handled disappear.
something else.

I relied on the first one so far which actually was a false assumption. But to implement that correctly I need to know which assumptions I can make that hold.
My preference would either be the workspace folder array and event correctness or handle and unhandle on the FS provider.

I think we should guarantee that all the workspace folders are accessible for extensions before any call to the file system provider is made. That way, when any request is made, the extension can see which folder the resource belongs to (we even have API for that afaik) and then make the correct adjustments to e.g. unzip the zip first for any request that is made.
Doing an artificial first stat call to the root seems very wrong to me, especially since this is a corner case for some extensions and not all and we would basically require a slow stat call for no real reason.

I think we should guarantee that all the workspace folders are accessible for extensions before any call to the file system provider is made.

💯

Workspace object will have all the root folders of the window, when the extension is activated. There is also an event when a root folder is added or removed. Not sure what is missing?

Not sure what is missing?

let's hope nothing

Closing based on that.

@sandy081 what was missing is the statement that no call is made to a FS provider before the add folder event is sent out. Otherwise a FS provider might not be able to resolve the root correctly. I think you got looped in because you make early calls to the FS provider from a configuration service.

@dbaeumer Got it. Yes you are right, when a workspace folder is added, I try to resolve its configuration too before triggering the event.

@sandy081 thanks for clarifying.

We closed this issue because we don't plan to address it in the foreseeable future. If you disagree and feel that this issue is crucial: we are happy to listen and to reconsider.
If you wonder what we are up to, please see our roadmap and issue reporting guidelines.
Thanks for your understanding, and happy coding!


Route traffic to a specific server when all weighted servers are down

Do you want to request a feature or report a bug?
Not sure. Traefik isn't matching my expectations as to how it should perform. But I guess whether it's a bug or feature request depends on how everyone else expects it to work.
Did you try using a 1.7.x configuration for the version 2.0?

[ ] Yes
[x] No

What did you do?

Set up two servers with wrr and health checks. One server having a weight of 1, the other a weight of 0.
What did you expect to see?
Requests should only be routed to the server with a weight of 1, assuming it's passing the health check. When it's failing the health check, requests should be routed only to the server with a weight of 0.
What did you see instead?
When the server with a weight of 1 is passing the health check, traffic is routing just to that server as I expected. However when the health check is failing, Traefik returns "Internal Server Error" and does not route traffic to the server with a weight of 0.
Output of traefik version: (What version of Traefik are you using?)

v1.7.10

What is your environment & configuration (arguments, toml, provider, platform, ...)?
# traefik.toml
defaultEntryPoints = ["http"]
AllowMinWeightZero = true

[entryPoints]
  [entryPoints.http]
  address = ":8000"

  [entryPoints.webadmin]
  address = ":8001"

[api]
  dashboard = true
  entryPoint = "webadmin"

[file]

[backends]
  [backends.srweb]
    [backends.srweb.healthcheck]
      path = "/Home/StatusCheck"
      interval = "10s"
    [backends.srweb.servers]
      [backends.srweb.servers.server0]
        url = 'https://stingray.localhost'
        weight = 1
      [backends.srweb.servers.maintenancepage]
        url = 'https://stingraymaintenance.stingray.localhost/'
        weight = 0

[frontends]
  [frontends.frontend1]
    entryPoints = ["http"]
    backend = "srweb"
    passHostHeader = false

If applicable, please paste the log output in DEBUG level (--logLevel=DEBUG switch)
WARN[2019-04-15T10:43:44-05:00] Health check failed: Remove from server list. Backend: "srweb" URL: "https://stingray.localhost" Reason: received error status code: 500
DEBU[2019-04-15T10:44:03-05:00] '500 Internal Server Error' caused by: all servers have 0 weight

What I am trying to achieve is similar to a Barracuda Last Resort Server. Basically, if all "real" servers are down (those with a weight > 1) then I want to route traffic to a special server so I can show a maintenance page. But I don't want the maintenance page showing up when any of the "real" servers have a passing health check.

Since you have AllowMinWeightZero set to true, it means that any server with a weight of 0 is removed from the list of available servers. So the behaviour that you are getting is the expected one.

@ljupchokotev Ah, I thought I had tested without that and not gotten the desired behavior. I found that setting by looking at some of the other issues. Do you know if the specific behavior for it is documented anywhere? It's not obvious to me what it means just from reading the name. Even when I get the --help from Traefik, the only description it gives is "Allow weight to take 0 as minimum real value." which just confuses me more.

You can find the documentation for AllowMinWeightZero here: https://docs.traefik.io/configuration/commons
Still, I don't think the workflow that you are looking for can be done in Traefik.

Ah. So I guess this is a feature request then.

Hi! I'm Træfiker :robot: the bot in charge of communication regulation.
Thanks for your interest in Traefik!
We dedicate the issue tracker to bug reports and feature requests only. My advanced AI tells me this one is neither of them.
I encourage your to seek community support: join our Slack workspace and reach out to us on the #support channel.
Another option is to use Stack Overflow and tag your question traefik.
You can of course double check Traefik's documentation :sweat_smile:

We dedicate the issue tracker to bug reports and feature requests only. My advanced AI tells me this one is neither of them.

Really? I find this incredibly frustrating. This is a feature request.


torch.aten.view to linalg

We are running into these issues when lowering torch.aten.view to linalg in these models (bart-large, mit-b0, t5-base, t5-large, ShuffleNet_v2_x2_0_vaiq_int8). Repro instructions can be found here:
bart-large.default.pytorch.torch.mlir:362:12: error: failed to legalize operation 'torch.aten.view' that was explicitly marked illegal
    %309 = torch.aten.reshape %281, %308 : !torch.vtensor<[?,9,1024],f32>, !torch.list<int> -> !torch.vtensor<[1,?,16,64],f32>
           ^
bart-large.default.pytorch.torch.mlir:362:12: note: see current operation: %834 = "torch.aten.view"(%832, %833) : (!torch.vtensor<[?,9,1024],f32>, !torch.list<int>) -> !torch.vtensor<[1,?,16,64],f32>

mit-b0.default.pytorch.torch.mlir:296:12: error: failed to legalize operation 'torch.aten.view' that was explicitly marked illegal
    %250 = torch.aten.reshape %247, %249 : !torch.vtensor<[?,?,?],f32>, !torch.list<int> -> !torch.vtensor<[1,32,128,128],f32>
           ^
mit-b0.default.pytorch.torch.mlir:296:12: note: see current operation: %938 = "torch.aten.view"(%936, %937) : (!torch.vtensor<[?,?,?],f32>, !torch.list<int>) -> !torch.vtensor<[1,32,128,128],f32>

t5-base.default.pytorch.torch.mlir:1860:13: error: failed to legalize operation 'torch.aten.view' that was explicitly marked illegal
    %1163 = torch.aten.tile %1155, %1162 : !torch.vtensor<[?,?,512],si64>, !torch.list<int> -> !torch.vtensor<[?,?,512],si64>
            ^
t5-base.default.pytorch.torch.mlir:1860:13: note: see current operation: %10542 = "torch.aten.view"(%10531, %10541) : (!torch.vtensor<[1,?,512,?,1,512],si64>, !torch.list<int>) -> !torch.vtensor<[?,512,?,1,512],si64>

t5-large.default.pytorch.torch.mlir:3480:13: error: failed to legalize operation 'torch.aten.view' that was explicitly marked illegal
    %2171 = torch.aten.tile %2163, %2170 : !torch.vtensor<[?,?,512],si64>, !torch.list<int> -> !torch.vtensor<[?,?,512],si64>
            ^
t5-large.default.pytorch.torch.mlir:3480:13: note: see current operation: %20514 = "torch.aten.view"(%20503, %20513) : (!torch.vtensor<[1,?,512,?,1,512],si64>, !torch.list<int>) -> !torch.vtensor<[?,512,?,1,512],si64>

torch-mlir-opt --convert-torch-onnx-to-torch  --torch-decompose-complex-ops --cse --canonicalize --convert-torch-to-linalg ./ShuffleNet_v2_x2_0_vaiq_int8.default.torch-onnx.mlir
./ShuffleNet_v2_x2_0_vaiq_int8.default.torch-onnx.mlir:364:12: error: failed to legalize operation 'torch.aten.view' that was explicitly marked illegal
    %360 = torch.operator "onnx.Reshape"(%344, %359) {torch.onnx.allowzero = 0 : si64} : (!torch.vtensor<[?,?,?,?],f32>, !torch.vtensor<[5],si64>) -> !torch.vtensor<[?,?,?,?,?],f32>
           ^
./ShuffleNet_v2_x2_0_vaiq_int8.default.torch-onnx.mlir:364:12: note: see current operation: %2002 = "torch.aten.view"(%1941, %1205) : (!torch.vtensor<[?,?,?,?],f32>, !torch.list<int>) -> !torch.vtensor<[?,?,?,?,?],f32>

Current status on above models: @kumardeepakamd @AmosLewis

bart-large:

 mlir::Value>, T = mlir::Value]: Assertion `isa<T>(*this) && "Invalid accessor called"' failed.
Please report issues to https://github.com/iree-org/iree/issues and include the crash backtrace.```
It's failing in fold-unit-dims pass.

2. **mit-b0**: iree-compile passed, [mismatch] issue.

3. **t5-base and t5-large**: both passed successfully.

4. **shufflenet_vaiq_int8**: iree-compile passed, [mismatch] issue.

Hi @Shukla-Gaurav, are you still working on this op? If not, then can you please update the issue?


Support non-SEPA payments as well

Specifying domestic attribute will suppress elements for SvcLvl code "SEPA". This is necessary for transactions within CH/LI, where SEPA transfers are refused. This is weird, IMHO, but specified in the Swiss Business Rules and observed by banks.

Is it possible to release a new version including this change?


fix: Change "writting" to "writing" (fix #564)

Please check if the PR fulfills these requirements

[x] It's the right issue type on the title
[x] When resolving a specific issue, it's referenced in the PR's title (e.g. fix #xxx[,#xxx], where "xxx" is the issue number)
[x] The commit message follows our guidelines
[x] Tests for the changes have been not added (for bug fixes/features) cause they are not needed in this case
[x] Docs have been added/updated (for bug fixes/features)
[x] It does not introduce a breaking change or has a description of the breaking change

Description
Change "writting" to "writing".

Thank you for your contribution to TOAST UI product. 🎉 😘 ✨

This fixes #564


Provide instructions of complete repo cloning via https

Description
During the current "reproducible build" debate in issue #188 and #192 I kept wondering if it is possible to enable a cloning of the repository via the https method. The instructions in the readme.md and reproducible.md just use ssh which may be a barrier for people who want to try building sparrow and haven't yet looked into ssh, don't have a Github account or want to test different build environments e.g in VMs without the hassle to transfer or add ssh-keys for those environments.
Actions taken
I tried git clone --recursive https://github.com/sparrowwallet/sparrow.git which works for all but the drongo submodule. The folder remains empty as a ssh connection fails to establish. In a short research I haven't found how to tackle the problem to include the submodule for https.
Solution
Add instructions in readme.md and reproducible.md for a complete https clone incl. the drongo submodule.

Fixed in 4f9b87b. Sparrow can now be recursively cloned with
git clone --recursive https://github.com/sparrowwallet/sparrow.git

as well as
git clone --recursive git@github.com:sparrowwallet/sparrow.git

README updated to reflect this in ca1f934.

It seems it still tries to clone via ssh into the drongo module. Not sure why it can't pull it with https?

note: I denied a further cloning although I have ssh keys. That's why the agent at the end isn't happy.

This is because you're checking out on a tag (1.5.0-beta1) which precedes the fix above. The next release will have the fix as a previous commit.

Aaaaah embarrassing, sorry for that. I should have given a thought or two before raising the question.


Where is the Spatial-Temporal Self-Attention?

Thanks for your works! I wonder where did you implement the Spatial-Temporal Self-Attention. In the appendix algorithm 1 and 2, only attention fusing is shown. And I also puzzled that would attention fusing conflict with Spatial-Temporal Self-Attention, because if attention fusing is performed afterSpatial-Temporal Self-Attention, it seems to be equivalent to not having performedSpatial-Temporal Self-Attention at all.

There are no conflicts because the technics of blending fusion and network architecture are basically orthogonal.
During inversion, we also inflate the original 2d self-attention to be spatial-temporal self-attention.
Then, we store the attention maps of spatial-temporal self-attention for each denoising step.
Finally, we fuse the stored attention map during editing.
You may check our implementation here, which is called during both inversion and denoising.

Ok, that's make sense! Thanks for your quick reply!


A few things I experienced

Hi Sid,
Hope all is going well with the new version.  I just tested a few things:
1) So because of how oauth-cache.R works, the .httr-oauth file gets saved in the directory where R is started from.  Would it be possible to add to authenticate.R a check to see if the .httr-oauth file exists in ~/.store/genomics_r_client/ and if not, then it would initiate the browser authentication and place a copy of the file in ~/.store/genomics_r_client/?  Otherwise depending on which directory I am in, then I will always have to authenticate via the browser if the file does not exist locally.
2) When I run the following getReadData query I get the following error, and it does not finish the request - the (...) are there to denote that there were many fetches of read data:
> getReadData(readsetId = "CPHG3MzoCRDRkqXzk7b6l_kB", chromosome ="1", start=100000000, end=101000000)
Fetching read data page
Parsing read data page
Continuing read query with the nextPageToken: EjBDaGhEVUVoSE0wMTZiME5TUkZKcmNWaDZhemRpTm14ZmEwSVNBVEVZOUtPUE1DZ0EYAiDM0JvNCw
...
Fetching read data page
Parsing read data page
Continuing read query with the nextPageToken: EjBDaGhEVUVoSE0wMTZiME5TUkZKcmNWaDZhemRpTm14ZmEwSVNBVEVZN2FlUE1DZ0EYAiDM0JvNCw
Fetching read data page
Parsing read data page
Continuing read query with the nextPageToken: EjBDaGhEVUVoSE0wMTZiME5TUkZKcmNWaDZhemRpTm14ZmEwSVNBVEVZOHF1UE1DZ0EYAiDM0JvNCw
Fetching read data page
Parsing read data page
Error in elementType(x) :
  error in evaluating the argument 'x' in selecting a method for function 'elementType': Error: evaluation nested too deeply: infinite recursion / options(expressions=)?
>

3) If I interrupt (via Ctrl-C) a getReadData request because I would like to start a different one, I can't seem to perform another request after that:
> getReadData(readsetId = "CMvnhpKTFhD04eLE-q2yxnU", chromosome ="1", start=100000000, end=101000000)
Fetching read data page
Parsing read data page
Continuing read query with the nextPageToken: Ei9DaGREVFhadWFIQkxWRVpvUkRBMFpVeEZMWEV5ZVhodVZSSUJNUmlQMU5jdktBRRgCIIGy8rQI
Fetching read data page
Parsing read data page
Continuing read query with the nextPageToken: Ei9DaGREVFhadWFIQkxWRVpvUkRBMFpVeEZMWEV5ZVhodVZSSUJNUmpxNU5jdktBQRgCIIGy8rQI
^C
>  getReadData(readsetId = "CMvnhpKTFhD04eLE-q2yxnU", chromosome ="1", start=10000000, end=101000000)
Fetching read data page
Error in function (type, msg, asError = TRUE)  :
  easy handled already used in multi handle

I would need to exit out of R, and resubmit the new getReadData request.
I know this is work in progress, but just wanted to give you a heads up.
Thanks,
Paul

Paul, thank you for this feedback.  We're breaking these down into smaller tasks.


Android version

Youtube is getting slowed down on mobile device too (at least for me) so i'd appreciate a version for Android mobile

https://github.com/ValdikSS/GoodbyeDPI?tab=readme-ov-file#similar-projects


Build error on warning

https://github.com/ARM-software/ComputeLibrary/blob/master/src/core/NEON/kernels/arm_conv/pooling/kernels/a64_s8_nhwc_avg_generic_depthfirst/generic.cpp#L87
rescale_value is of type in32_t so even it is cast to int64_t , it will never be equal to 1ll<<31
src/core/NEON/kernels/arm_conv/pooling/kernels/a64_s8_nhwc_avg_generic_depthfirst/generic.cpp: In function ‘void arm_conv::pooling::a64_s8_nhwc_avg_generic_depthfirst_impl(uint64_t, uint64_t, uint64_t, const int8_t* const*, int8_t*)’:
src/core/NEON/kernels/arm_conv/pooling/kernels/a64_s8_nhwc_avg_generic_depthfirst/generic.cpp:88:45: error: comparison is always false due to limited range of data type [-Werror=type-limits]
if (static_cast<int64_t>(rescale_value) == (1ll << 31))
^
cc1plus: all warnings being treated as errors
scons: *** [build/src/core/NEON/kernels/arm_conv/pooling/kernels/a64_s8_nhwc_avg_generic_depthfirst/generic.o] Error 1

Hi @ek9852
Would you please share the output of aarch64-linux-gnu-g++ -v and the build command you use to compile the library?

@morgolock  I just use this ARM provided dockerfile
https://github.com/ARM-software/armnn/blob/branches/armnn_21_02/docker/x86_64/Dockerfile#L147
But as I said, it is cleanly     if (static_cast<int64_t>(rescale_value) == (1ll << 31)) will always false
since rescale_value is of type int32_t.

Hi @ek9852
This patch fixes the problem: https://review.mlplatform.org/c/ml/ComputeLibrary/+/5741
Thanks for reporting this.


"Can only call GUI functions from inside OnGUI" error

Using only Amand's Graphics and no other mods.
Game will load with no issues, chams will still show. upon pressing the hot key for the aim bot the following error is produced,
2023-04-18 14:37:49.475 -06:00|<IP_ADDRESS>.22617|Error|Default|
EXCEPTION: System.TypeInitializationException: The type initializer for 'EFT.Trainer.UI.Render' threw an exception. ---> System.ArgumentException: You can only call GUI functions from inside OnGUI.
at UnityEngine.GUIUtility.CheckOnGUI () [0x00010] in :0
at UnityEngine.GUI.get_skin () [0x00001] in :0
at EFT.Trainer.UI.Render..cctor () [0x00000] in :0
--- End of inner exception stack trace ---
at EFT.Trainer.Features.Aimbot.IsInFieldOfView (UnityEngine.Vector2 screenPosition) [0x0000f] in :0
at EFT.Trainer.Features.Aimbot.TryGetNearestTarget (EFT.Player& localPlayer, UnityEngine.Camera& camera, UnityEngine.Transform& nearestTarget) [0x000d8] in :0
at EFT.Trainer.Features.Aimbot.UpdateWhenHold () [0x00000] in :0
at EFT.Trainer.Features.HoldFeature.Update () [0x00015] in :0
at EFT.Trainer.Features.Aimbot.Update () [0x00000] in :0
2023-04-18 14:37:49.475 -06:00|<IP_ADDRESS>.22617|Error|Default|ArgumentException: You can only call GUI functions from inside OnGUI.
UnityEngine.GUIUtility.CheckOnGUI () (at :0)
UnityEngine.GUI.get_skin () (at :0)
EFT.Trainer.UI.Render..cctor () (at :0)
Rethrow as TypeInitializationException: The type initializer for 'EFT.Trainer.UI.Render' threw an exception.
EFT.Trainer.Features.Aimbot.IsInFieldOfView (UnityEngine.Vector2 screenPosition) (at :0)
EFT.Trainer.Features.Aimbot.TryGetNearestTarget (EFT.Player& localPlayer, UnityEngine.Camera& camera, UnityEngine.Transform& nearestTarget) (at :0)
EFT.Trainer.Features.Aimbot.UpdateWhenHold () (at :0)
EFT.Trainer.Features.HoldFeature.Update () (at :0)
EFT.Trainer.Features.Aimbot.Update () (at :0)
UnityEngine.DebugLogHandler:LogException(Exception, Object)
Class311:UnityEngine.ILogHandler.LogException(Exception, Object)
UnityEngine.Debug:CallOverridenDebugHandler(Exception, Object)
also throws the same error when attempting to draw the locations of tracked items

I just found that by having a tracked item or exfils on from the main menue prior to loading into a raid, the trainer works as intended with no errors, its only if nothing is being drawn by the trainer then load into a raid and attempt to use the aimbot or track items that it errors out

I see, this one is tricky.
Aimbot.TryGetNearestTarget is used in the Update loop, and is using Render.ScreenCenter. The very first time, accessing the Render type when nothing is initialized, this will trigger the static ctor to initialize all static fields, including StringStyle but given it is using the Gui API, it will fail because we are not in the OnGui loop.
At the opposite when we rendered anything first (I suppose you don't even enable the HUD), the initialization is called properly.

We should be good with that:
https://github.com/sailro/EscapeFromTarkov-Trainer/commit/ed8babfc2a27a6d0f0086aeed3cc9f8b2f9547ce
Please re-run the installer to get the fix.

fantastic thanks for the quick fix!


Release prep for 5.3.7

Release note and versioning updates where needed

I keep putting my comments on the wrong files. All this generated documentation is confusing me. I'll get it right eventually.


Merging Instances When Accesses Come from Different Unroll Paths

For example, in the case from KMP:
      val global_matches = 'UNROLLED.Sequential.Reduce(Reg[Int](0))(... par 4) {chunk =>
        ...
        val q = Reg[Int](0)
        'LCASeq.Sequential.Foreach(...) { i =>
          FSM(0)(state => state != 1) { state =>
            if ((q > 0) && (string_sram(i) != pattern_sram(q))) q := kmp_next(q)  // usage 1
          }{state => mux((q > 0) && (string_sram(i) != pattern_sram(q)), 0, 1)}  // usage 2
          Pipe{if (pattern_sram(q) == string_sram(i)) { q :+= 1 }}                        // usage 3
          ...
          }
        }
        ...
      }{_+_}

It is merging usage 1 from access unroll {1,0} with usage 3 from access unroll {2,0} for pattern_sram.  It sees the LCA as LCASeq and therefore thinks they don't require concurrent access since this is a sequential outer. But after unrolling, the LCA actually becomes the parallel that binds the unrolled loops of UNROLLED

LCAParallel is an app that tests this.  We now look for how accesses unroll relative to each other and the lockstepness of them.  Without lockstep, we still group the accesses together but in ExhaustiveBanking, we substitute iterators based on their unroll path and try to bank that way.  If this fails, we then regroup based on unroll path and create a duplicate for each path that gets banked on its own.


feat: add aws directory service support

Objective:
Add support for aws directory service: Fixes #441
Pricing details:
Directory service cost output based on:

Directory type and size/edition
Monthly data transfer out to an external region from the directory
Shared directory usage (only applicable to Microsoft AD)

 aws_directory_service_directory.microsoft_ad_standard
 └─ Directory service (Microsoft AD, Standard)                     730  hours           $43.80

 aws_directory_service_directory.microsoft_ad_with_usage
 ├─ Directory service (Microsoft AD, Standard)                     730  hours           $43.80
 ├─ Outbound data transfer (from us-east-1, to eu-west-2)        2,000  GB              $40.00
 └─ Directory sharing                                                8  accounts         $0.14

Status:

[x] Added to resource_registry.go
[x] Added internal/resources file
[x] Added internal/provider/terraform/.../resources file
[x] Added usage parameters to infracost-usage-example.yml
[x] Added test cases without usage-file
[x] Added test cases with usage-file
[x] Compared test case output to cloud cost calculator.
[x] Created a PR to update "Supported Resources" in the docs)

Issues:
I was unable to find a outbound data transfer pricing scoped to the AWSDirectoryService service. So I have used AWSDataTransfer service which seems to align with the pricing shown on the pricing page. Let me know if you think this is ok or you manage to find a more specific costing in the pricing db.

Regarding feedback on the new add resource instructions: I found them useful to get going, but then just dove into code after the fact. I watched the first bit of the video but then dropped off as I had seen previous video with Ali. I think this is a good addition though.  In general I think the docs are pretty good, especially around specific gotchas like updating tests. Things that I think the docs could improve on:

I think it could be easier to show in the docs the actual placement of the files with a simple directory tree output. The naming's of the files are so similar I slipped up on first placement.
It was a little unclear at first what the mandatory fields on the example resource meant. I initially thought that these were mandatory for all resources but then realised when I got stuck in that these were just fields that were parsed from a provider and thus changed per resource.
I think better docs in the code could a good way to help new contributors. I'm going to write an issue for this as I think there's a bit too much to dig into with this PR.

Don't even show show data transfer costs for this resource. It's the same data transfer prices as normal, and for other resources we exclude them since the user can just add them to the aws_data_transfer usage resource. So maybe this is something we need to fix in a more generic way across resources to ensure they are consistent.

+1 on this option, I think data transfer will need more investigation, e.g. can we do lookups from CloudWatch for different resources?

@aliscott I have resolved all comments, build seems to be failing due to github outage atm: https://www.githubstatus.com/incidents/m78sypncxsdw

👍 thanks! I'll re-run the build when GitHub is back.