Add an option to NextJS to set it up for github pages

Also includes the change here given a dependency

I added some commits with bug fixes and to get Windows to play nicely. I have tested both Ubuntu and Window platforms as of f4bf4e20b39dca95f26978ae9bd4f0c603377194 and everything is working on that end! Also confirmed the deployed web pages from Github Pages is working correctly.


Connection reuse returning no error, but empty results

I'm not very familiar with Go, but rewrote my api to use go-redis, so please excuse me if this is a simple mistake.
On my first attempt, I put the NewClient variable in my handler function outside of main, which works well aside from being super inefficient and using thousands of file descriptors.
Realizing my mistake, I moved it to the main function with a global var.  Now it works with reusing connections, but randomly it will start returning empty results with no errors!?   Which of course is horrible, because that registers as a valid result. I can force it to throw errors, and have gotten EOF errors in some cases with timeouts, but this has no error.   It happens whether I use Sentinel or my keepalived ip.  The only reliable method is to connect everytime handler function is run.
Here is a basic outline of the program.  Same
import (
	redis "github.com/go-redis/redis/v8"
)
var rdb *redis.Client
var ctx = context.Background()
func main() {
	router := http.NewServeMux()
	router.HandleFunc("/api", ApiHandler)
	srv := &http.Server{
		Addr:    ":8080",
		ReadTimeout:  5 * time.Second,
		WriteTimeout: 10 * time.Second,
	}
	rdb = redis.NewClient(&redis.Options{
			Addr: "<IP_ADDRESS>:6379",
			Password: "password",
			DB:       0,  // use default DB
			//DialTimeout:        10 * time.Second,
			//ReadTimeout:        30 * time.Second,
			//WriteTimeout:       30 * time.Second,
			PoolSize: 1000,
			//PoolTimeout:        30 * time.Second,
			//IdleTimeout:        time.Second * 25,
			IdleCheckFrequency: time.Second * 5,
			MaxRetries:         3,
	})
	log.Fatal(srv.ListenAndServe())
}
func ApiHandler(w http.ResponseWriter, r *http.Request) {
	key, ok := r.URL.Query()["key"]
	api_key := key[0]
	//works if I move rdb newclient here
	key_info, err := rdb.HGetAll(ctx, "key:" + api_key).Result()
	if err != nil {
		log.Println("key info error: ", err)
	} else  len(key_info) == 0 {
		log.Println("key info does not exist: ", err)
		log.Println("Missing apiKey is: " + api_key)
	} else {
		log.Println("Good apiKey is: " + api_key)
	}
	defer r.Body.Close()
	return
}

After many connections, the same key that registers as Good will randomly come up as Missing.  Sometimes once, sometimes many times, before stating Good again.   If I move the NewClient to my comment in ApiHandler, it is always Good.
Thanks!

It may be that the interval between multiple tests is too long (http), which causes the active connections in the connection pool to be closed by the server. We have recently added a new function to check whether the connection is healthy, and a new version will be released later. You can try it at that time

Thanks for the reply!  I think what happens is that sometimes multiple connections are started in the pool, and then at slower periods they are closed when only one is needed.
Either way, shouldn't it be throwing an error though?  Otherwise I can't really trust an empty or 0 result and would need to ping the server to make sure, correct?
Appreciate all the work!

I can't replicate your environment, can you make an example that triggers an empty result?  (err=nil)

I executed the above example, and I did not find any empty results.

Good news, I'm dumb.
I knew I screwed up when my Ping test came back good, but results were still empty.  Then I realized my synchronous loving brain didn't grasp that in my full program, if I make a db change with rdb.Do(Select ,1), that the change would effect other requests running at the same time.  Doh!  Now I've made separate NewClients for each db connection, and all seems to be working well.  Not sure if there's a better way to do it, but having a couple clients isn't a problem seeing as I had 1000s the other way.
Thanks for the help!

OK, then we close this issue.


Use lang pack date formats

As part of https://github.com/tastyigniter/TastyIgniter/pull/605

Looking at the changes here, I don't think you should use the convert_php_to_moment_js_format helper... converting l, jS F Y \a\t h:i a to DD MMM \a\t HH:mm will be a tricky and could have unexpected results... I'd say duplicate the lang string under main namespace for the components? ex. main::lang.date_format..

So for each component date string have a new lang pack entry? Isn’t that just the same problem moved to a different place - ie that there are so many date strings to update?
I’m happy to spend some time getting the helper function working ‘right’?

No no, all of the component's date formats will be pulled from the main:: language strings and for the backend from admin:: (you can move them from the system:: lang file to the admin lang file since we now have different formats for frontend and backend). So it's just duplicating the same lang strings from the system into main lang file

So front end uses moment format, and backend PHP? Or just different PHP for front end and back end>

Exactly!

Moment format for the frontend and PHP for backend.

Should be all done

Yups, I'll merge all the PR later.. Thank you!
Also, I've put all the lang string in one place under the system namespace, didn't want to bother you again with the changes ;).

Thats cool thanks 👍


Publish the latest changes to the npm

I'd like to have all the latest updates via npm.
Could you follow the rule to publish all changes to the npm?
It's really easy with the grunt-release

This seems like a good thing to automate to make sure we don't forget it.  But we're up to date right now, I believe - latest release is 1.1.0, and that's the version in npm.

npm install<EMAIL_ADDRESS>find node_modules/polyfill-service/polyfills/ -name fetch

There's no the fetch method

We haven't made a new release since fetch was added.  A release is not as simple as just tagging and pushing to npm - we need release notes, fair warning to users and a QA period on qa.polyfill.io.  I know you're keen to have the fetch polyfill.  Bear with us, it should be out this week!

Got it, thanks!


Added system check benchmark

As requested by @charettes on Django Ticket #35246 for covering my recent improvements to system check performance.
I ran the benchmark locally on recent commits and got this nice graph out:

The precipitous drop is due to https://github.com/django/django/commit/28a3fbe0048883fdd5cefd6ffecb88e351121891 , which fixed O(n) execution of the URL custom error handler checks.
The benchmark environment doesn’t run every system check because it doesn’t install all contrib apps, notably the admin, and doesn’t use some features like GenericForeignKey. It also doesn’t have enough models, model inheritance, or relations to trigger some of the slownesses I’ve worked on. But I hope this benchmark is a good start.

Thanks a ton @adamchainz, I think it's fine if it doesn't cover everything after all most projects don't either.

Cheers @smithdc1 ! It seems there’s no way to trigger benchmarks on GHA for the previous commits, so the improvements so far won’t be recorded outside my screenshot above. But at least we have a defense against future regressions :)

It seems there’s no way to trigger benchmarks on GHA for the previous commits

Yes, this is a bit of a shame, refs: https://github.com/django/django-asv/issues/47

But at least we have a defense against future regressions :)

Future regressions (even small ones) should be easier to track going forward. Something changed which means running them in GHA is now far more repeatable than before. https://django.github.io/django-asv/#req_resp_benchmarks.http_methods.benchmark.HttpMethods.time_get_method.
I wonder if the benchmark runner on django/django will also be more reliable now because of that 🤔


Rewrite

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@9a7e670). Click here to learn what that means.
The diff coverage is 100%.

@@           Coverage Diff           @@
##             master    #12   +/-   ##
=======================================
  Coverage          ?   100%
=======================================
  Files             ?     15
  Lines             ?    222
  Branches          ?     43
=======================================
  Hits              ?    222
  Misses            ?      0
  Partials          ?      0

Impacted Files
Coverage Δ

src/WebpackLicensePlugin.ts
100% <100%> (ø)

src/ModuleDirectoryLocator.ts
100% <100%> (ø)

src/WebpackFileSystem.ts
100% <100%> (ø)

src/index.ts
100% <100%> (ø)

src/LicenseMetaAggregator.ts
100% <100%> (ø)

src/WebpackChunkIterator.ts
100% <100%> (ø)

src/PackageJsonReader.ts
100% <100%> (ø)

src/WebpackAssetManager.ts
100% <100%> (ø)

src/WebpackChunkModuleIterator.ts
100% <100%> (ø)

src/LicenseTextReader.ts
100% <100%> (ø)

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9a7e670...c17fb6a. Read the comment docs.


Demo/analyzer policy rules

Add workflow rule engine to analyzer demo

Hello @thisismatu got an email for this PR but now I don't see myself as reviewer here. Let me know when you need help with merging the PR.

@bigdatabaracus preview link https://workflow-demo.vercel.app/


pad/pdn: improve bump handling

Adds:

dont_route to assign_io_bump command to allow for assignment of bumps without attempting to route the bump
-bump flag to pdn macro grid definition to skip adding grids to bumps that have already been routed to.

What is the use case? Pre routing?

What is the use case? Pre routing?

The bumps in the center of chip cannot be routed with the RDL router. This still allows them to be correctly assigned (ie. a bpin is created and the bump pins are connected to the correct net), but the RDL router will not attempt to route them. PDNgen can then add a grid around them to connect to the bump properly.


Layout issues in ColorPicker UI

In the UI for the ColorPicker, in the panel with the Xoetrope color wheel, one of the text boxes does not have enough height to display the text.  (It is the textbox with the hexadecimal color value.  May depend on font and Windows display settings.  I use the Display setting "Medium 125%".)
There is also a tiny button "reset the color wheel saturation and brightness" that doesn't seem to do anything.
In the panel with RGB sliders, there should probably be a small gap added between the sliders and the text boxes.

Do you mind attaching a screenshot of how it looks like under that display setting?

I'll replace the reset button (that looks like a checkbox) with a circular arrow icon. The purpose of this control is to reset the color wheel state after using CTRL+drag on one or both of the sliders. Ctrl-dragging changes the saturation and / or brightness of the entire wheel, and that reset brings it back to full values.

Ok, thanks.  I didn't know about the Ctrl-dragging so it looked like that button had no effect.
I've also noticed that if you right-click in the colored field just below "Use web safe ..." a pop-up will list colors related to the current Substance skin.  While that is useful for programmers, it probably isn't useful for end users of programs created with Substance.  Whether you want to remove that or not is totally up to you.

The list of "control" colors shown when you click on that colored label is intentional by the original author of the code. Other issues should be addressed with the latest drop of 8.0.0-dev.


Enable users to update the  max HTLC for a given channel(s).

Depends on #2437 (which depends on #2434, which depends on #2296). Partially closes #2182.
In the previous PRs, we implement parsing, storing and re-sending the new max HTLC field that has been added to ChannelUpdates. We also consider the max HTLC during pathfinding.
In this PR, we enable users to update the max HTLC for a given channel or channels.
Edit 08/6/2019: currently it's a mandatory field for UpdateChannelPolicy, gonna edit to make it optional.
Edit 2: done
Edit 8/30/19: depends on #3439 to resolve this comment and similar.

Needs rebase.

Needs a rebase!

Picking up this PR in #3523


Un-shortening

Hi Ivan
Great new feature with Un-shortening. Could you add eur02.safelinks.protection.outlook.com? Their might be more version so might be that you only need to look for protection.outlook.com.
Best Mads

Thanks! These URLs are not entirely shortened, the actual link is embedded within URL.
The issue with un-mangling it is you might not want to do it as the link goes via Microsoft antivirus check first. So the rule has to be applied to un-mangled URL, but you really want to open the "safelinks" one.
I'll have to make some changes to implement this, which would be really useful.

It would be really helpful if this was implemented. Below I did a test link for https://www.google.com, so you have a full example. I have replaced some of the info, but it still align to how the link would look.
Encoded:
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2F%26data%3D05|01|YYY%40XXX.com|e0db85sd428csdsdba492a6f7|8f9b88sdse4be3aae420sds42306|1|0|6382sds448699|Unknown|TWFasdsasdwMDAiLCJQIjoiV2luMzIiLCsdshaWwiLCJXVCI6Mn0%3D|3000|||%26sdata%3DmxTjSxm%2BOJmi3ulKDsDdsdsG07eB3IYt2McJY21U%3D%26reserved%3D0
Decoded:
<EMAIL_ADDRESS>

Relevant documentation section: https://learn.microsoft.com/en-us/microsoft-365/security/office-365-security/safe-links-about?view=o365-worldwide

Amazing work, thanks for this!

Amazing work, thanks for this!


Caught trace is less than expected for nested requests

When there's chain of requests (one calling into another request), the caught trace is way less than expected.
This is caused by the algorithm in the request event matching. The fix will be fixed in 2.0.0-beta4.

This is fixed in 2.0.0-beta4.


[API 8] Refactoring of LaunchProjectileEvent - Removal of TargetProjectileEvent

SpongeAPI | SpongeCommon
This is a breaking change.
This PR removes the TargetProjectileEvent and changes the LaunchProjectileEvent.
Now, the event is an AffectEntityEvent: the point is to make the event handle several projectiles at the same time, which is possible in mods (like the Thermal Expansion Multishot enchantment).

Note: This PR was automatically closed because the bleeding branch was deleted. It will need to be reopened to target api-8 instead.  See https://github.com/SpongePowered/SpongeAPI/pull/1428#issuecomment-539689855.


Transform request body

Hi guys,
congrats for this awesome lib.
I need to transform the request body before it has been proxied.
This is my code
function write(chunk, encoding, next){
  var transformed_body = new BodyHelper(JSON.parse(chunk), this.body_transformation).apply()
  this.push(JSON.stringify(transformed_body))
  next();
}

var transform = through.obj(write);
transform.body_transformation = // custom rule
req.pipe(transform).pipe(request.post(settings)).pipe(res)

In this example the object transformed_body is correctly formatted, but the following piping not receive the body correctly.
Instead even if I use .obj function I need to stringify the object because it throw the error Argument passed must be a string or buffer, why?
Where I'm making a mistake?
Thanks

Are you sure the error is from the transform stream, and not from the others it's piped to?
At which line exactly do you get the error?
Do you have a testable example?

If I remove the transformation pipe all work fine.
With the transformation seems like the request module (https://github.com/request/request) it not able to copy the body

transform.body_transformation = // custom rule is not implemented by through2 thus will result in a noop. Try updating your write function or implementing multiple through streams.

@yoshuawuyts I use transform.body_transformation just to set an object in order to find it in the context

oh right, I missed that. Don't do that, it's confusing.

It's not clear for me why this code doesn't works
function write(chunk, encoding, next){
  next(null, chunk);
}
var transform = through.obj(write);
req.pipe(transform).pipe(res)

In this example I expected that the response's body is the same of the request one because the transformation is null. Why the response body is empty?

This does work, you're probably having issues in other parts of your code. Here's a minimal http example:
const through = require('through2')
const http = require('http')

http.createServer(function (req, res) {
  req.pipe(through((chunk, enc, next) => next(null, chunk))).pipe(res)
}).listen(8080)

$ curl localhost:8080 -d hello-world
hello-world

I hope this helps you narrow down your issue. Cheers!


scipy.special.chebyt(N) throw a 'TypeError' when N > 64

I am using Anaconda 32 bit on windows 7 64 bit.
test code:
from scipy.special import chebyt
import scipy as scp
print scp.__version__

for i in range(63,200):
    print i
    print chebyt(i)(0.2)

output:
0.17.1
63
-0.118896470865
64
0.949066574424
65
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-243-24c6fbe0b8bd> in <module>()
      5 for i in range(63,200):
      6     print i
----> 7     print chebyt(i)(0.2)

C:\Anaconda2\lib\site-packages\scipy\special\orthogonal.pyc in chebyt(n, monic)
   1201     kn = 2**(n - 1)
   1202     p = orthopoly1d(x, w, hn, kn, wfunc, (-1, 1), monic,
-> 1203                     lambda x: eval_chebyt(n, x))
   1204     return p
   1205

C:\Anaconda2\lib\site-packages\scipy\special\orthogonal.pyc in __init__(self, roots, weights, hn, kn, wfunc, limits, monic, eval_func)
    143             kn = 1.0
    144         self.__dict__['normcoef'] = mu
--> 145         self.__dict__['coeffs'] *= kn
    146
    147         # Note: eval_func will be discarded on arithmetic

TypeError: ufunc 'multiply' output (typecode 'O') could not be coerced to provided output parameter (typecode 'd') according to the casting rule ''same_kind''

Hopefully fixed in #6723. Note, however, that there's a reasonable chance you don't want to be doing that anyway--if you just want to evaluate Chebyshev polynomials use eval_chebyt directly. If you want to work with a Chebyshev series use numpy.polynomial.chebyshev. If you actually want the coefficients of the 65-200th Chebyshev polynomials then chebyt probably isn't giving very accurate results anyway.


Updates: Logged out user, anchor point on comments

Who is your user?
Logged out users
What are they trying to achieve?
Read an update
When a logged out user is linked to an update, I want them to land at the beginning of the update, so I can incentivise them to read it
How are they currently doing this?
Currently the anchor point is set to the log-in to make a comment section

How well understood is this problem?

[ ] Surfacing: if you feel this is a seed of an idea that needs to be researched.
[ ] Understanding: if you've done some research and feel it is already well understood and ready for shaping.
[ ] Shaping:  if you feel a solution is clearly defined and can be made into a pitch (read more about shaping)
[ ] Pitching: if you feel that it is ready to be presented for prioritisation (read more about pitching)

I was able to reproduce, here for example: https://opencollective.com/opencollective/updates/authentication-with-password-now-available. It likely originates from the autofocus on the sign-in form, which we should disable on this page since signing in is not its primary intent.


Convert .. code-block:: in the docstrings to .. doctest:: or .. test-code::

Python code blocks  (e.g., .. code-block:: python or .. code:: python) in the docstrings are not tested and can easily become out of date. Instead, they should use doctest, which can be done via .. doctest:: or .. test-code::. Any test setup (that should not be printed in the documentation but needs to run to generate necessary fixtures) can be in a .. test-setup:: block or the global docs/source/doctest_fixtures.py file.

Closed as mostly done.


Getting Null Value of NAD during EDIFACT deserialization

Hi @cleftheris ,
I'm trying to deserialize the EDIFACT file, but the value of NAD is NULL.
This is my code of model:
    public class ORDERS
    {
        public MES MES { get; set; }
    }

    [EdiMessage]
    public class MES
    {
        public RFF RFF { get; set; }
        public NAD NAD { get; set; }
    }

    [EdiSegmentGroup("RFF", SequenceEnd = "NAD")]
    public class RFF
    {
        [EdiCondition("ADE", Path = "RFF/0")]
        public RFF_Element AccountNumber { get; set; }

        [EdiCondition("PD", Path = "RFF/0")]
        public RFF_Element PromotionDealNumber { get; set; }
    }

    [EdiElement, EdiPath("RFF/0")]
    public class RFF_Element
    {
        [EdiValue("X(3)", Path = "RFF/0")]
        public string ReferenceQualifier { get; set; }

        [EdiValue("X(35)", Path = "RFF/0/1")]
        public string ReferenceNumber { get; set; }
    }

    [EdiSegmentGroup("NAD", SequenceEnd = "CUX")]
    public class NAD
    {
        [EdiCondition("BY", Path = "NAD/0")]
        public NAD_Element Buyer { get; set; }

        [EdiCondition("SU", Path = "NAD/0")]
        public NAD_Element Supplier { get; set; }

        [EdiCondition("DP", Path = "NAD/0")]
        public NAD_DeliveryParty DeliveryParty { get; set; }

        [EdiCondition("VA", Path = "RFF/0")]
        public RFF_Element VATRegistrationNumber { get; set; }
    }

    [EdiElement, EdiPath("NAD/0")]
    public class NAD_Element
    {
        [EdiValue("X(3)", Path = "NAD/0")]
        public string PartyQualifier { get; set; }

        [EdiValue("X(35)", Path = "NAD/1")]
        public string PartyIdentification { get; set; }

        [EdiValue("X(3)", Path = "NAD/1/2")]
        public string ResponsibleAgency { get; set; }
    }

    [EdiElement, EdiPath("NAD/0")]
    public class NAD_DeliveryParty
    {
        [EdiValue("X(3)", Path = "NAD/0")]
        public string PartyQualifier { get; set; }

        [EdiValue("X(35)", Path = "NAD/1")]
        public string PartyIdentification { get; set; }

        [EdiValue("X(3)", Path = "NAD/1/2")]
        public string ResponsibleAgency { get; set; }

        [EdiValue("X(3)", Path = "NAD/8")]
        public string Country { get; set; }
    }

And this is the result:

There is the EDIFACT file:

RFF+ADE:1234'
RFF+PD:1704'
NAD+BY+1234567890::9'
NAD+SU+1234567890::9'
NAD+DP+1234567890::9+++++++GB'
RFF+VA:GB727255821'

The value of RFF is correct, but the value of NAD is NULL.
And another question is, I want to serialize POCO to EDIFACT without outputting UNA segment. How can I do this?
Do you have any suggestion for me?
Thank you very much!
Nick

Hi @stidiat did you manage to resolve the issue?
I can see that for NAD you are using an EdiElement attribute that should only contain one element - for example the NAD/1 but instead you are using it as if it was decorated as a Segment. In your case you should be using one segment class for NAD and an other to define the segment group. The group must inherit from the same class that defines the starting segment NAD in your case. Also it does not seem right to be using the SegmentGroupAttribute in your case . In order to see more examples of SegmentGroup use cases checkout other questions regarding loops and segment groups
1. One way todo this
   public class ORDERS
    {
        public MES MES { get; set; }
    }

    [EdiMessage]
    public class MES
    {
        [EdiCondition("ADE", Path = "RFF/0")]
        public RFF AccountNumber { get; set; }

        [EdiCondition("PD", Path = "RFF/0")]
        public RFF PromotionDealNumber { get; set; }

        [EdiCondition("BY", Path = "NAD/0")]
        public NAD Buyer { get; set; }

        [EdiCondition("SU", Path = "NAD/0")]
        public NAD Supplier { get; set; }

        [EdiCondition("DP", Path = "NAD/0")]
        public NAD DeliveryParty { get; set; }

        [EdiCondition("VA", Path = "RFF/0")]
        public RFF VATRegistrationNumber { get; set; }
    }
   [EdiSegment, EdiPath("RFF")]
    public class RFF_Element
    {
        [EdiValue("X(3)", Path = "RFF/0")]
        public string ReferenceQualifier { get; set; }

        [EdiValue("X(35)", Path = "RFF/0/1")]
        public string ReferenceNumber { get; set; }
    }
    [EdiSegment, EdiPath("NAD")]
    public class NAD
    {
        [EdiValue("X(3)", Path = "NAD/0")]
        public string PartyQualifier { get; set; }

        [EdiValue("X(35)", Path = "NAD/1")]
        public string PartyIdentification { get; set; }

        [EdiValue("X(3)", Path = "NAD/1/2")]
        public string ResponsibleAgency { get; set; }

        [EdiValue("X(3)", Path = "NAD/8")]
        public string Country { get; set; }
    }

2. And another way
   public class ORDERS
    {
        public MES MES { get; set; }
    }

    [EdiMessage]
    public class MES
    {
        public List<RFF> ReferenceNumbers { get; set; }
        public List<NAD> Parties { get; set; }
    }

   [EdiSegment, EdiPath("RFF")]
    public class RFF_Element
    {
        [EdiValue("X(3)", Path = "RFF/0")]
        public string ReferenceQualifier { get; set; }

        [EdiValue("X(35)", Path = "RFF/0/1")]
        public string ReferenceNumber { get; set; }
    }
    [EdiSegment, EdiPath("NAD")]
    public class NAD
    {
        [EdiValue("X(3)", Path = "NAD/0")]
        public string PartyQualifier { get; set; }

        [EdiValue("X(35)", Path = "NAD/1")]
        public string PartyIdentification { get; set; }

        [EdiValue("X(3)", Path = "NAD/1/2")]
        public string ResponsibleAgency { get; set; }

        [EdiValue("X(3)", Path = "NAD/8")]
        public string Country { get; set; }
    }

Hi @cleftheris , I'm very appreciate for your reply.
Yes, I managed to do it in another way.
The point is that I need to output it in the original order, this is the document:

And this is my code:
public class ORDERS
{
    public MES MES { get; set; }
}

[EdiMessage]
public class MES
{
    [EdiCondition("ADE", Path = "RFF/0")]
    public RFF AccountNumber { get; set; }

    [EdiCondition("PD", Path = "RFF/0")]
    public RFF PromotionDealNumber { get; set; }

    [EdiCondition("BY", Path = "NAD/0")]
    public NAD Buyer { get; set; }

    [EdiCondition("SU", Path = "NAD/0")]
    public NAD Supplier { get; set; }

    [EdiCondition("DP", Path = "NAD/0")]
    public NAD DeliveryParty { get; set; }

    [EdiCondition("VA", Path = "RFF/0")]
    public RFF VATRegistrationNumber { get; set; }
}
[EdiSegment, EdiPath("RFF")]
public class RFF
{
    [EdiValue("X(3)", Path = "RFF/0")]
    public string ReferenceQualifier { get; set; }

    [EdiValue("X(35)", Path = "RFF/0/1")]
    public string ReferenceNumber { get; set; }
}
[EdiSegment, EdiPath("NAD")]
public class NAD
{
    [EdiValue("X(3)", Path = "NAD/0")]
    public string PartyQualifier { get; set; }

    [EdiValue("X(35)", Path = "NAD/1")]
    public string PartyIdentification { get; set; }

    [EdiValue("X(3)", Path = "NAD/1/2")]
    public string ResponsibleAgency { get; set; }

    [EdiValue("X(3)", Path = "NAD/8")]
    public string Country { get; set; }
}

I know the code is not up to standard or robust. But it can output the text in original order like below:

RFF+ADE:1234'
RFF+PD:1704'
NAD+BY+1234567890::9'
NAD+SU+1234567890::9'
NAD+DP+1234567890::9+++++++GB'
RFF+VA:GB727255821'

I think I will rewrite the file in your standard way later. Because there are very complicated parts in the document(EDIFACT D96A INVOIC) like this:

Thank you again for your reply.
Best regards,
Nick


POM, BOM updates

Picking up https://github.com/jenkinsci/bom/pull/970 & https://github.com/jenkinsci/bom/pull/977. May fail due to https://github.com/jenkins-infra/helpdesk/issues/2864.

Not just slow but broken:
Connect to repo.jenkins-ci.org:443 [repo.jenkins-ci.org/<IP_ADDRESS>] failed: Connection timed out: connect


LFADS Tutorial running error - unhashable type of init_params

I got an error when running LFADS tutorial codes using integrator RNN inputs.

I found "init_params" is a dict type.
But I thought when the code generating "init_params", it's already dict type.
Anyone knows how to fix this?

I got an error when running LFADS tutorial codes using integrator RNN inputs.

I found "init_params" is a dict type.
But I thought when the code generating "init_params", it's already dict type.
Anyone knows how to fix this?

im using :
cudatoolkit               11.3.1
cudnn                     8.2.1
jax                       0.2.20
jaxlib                    0.1.71+cuda111

Thanks @ecosan327 -- I had this issue after a jax version update a while ago. One solution is to use FrozenDict of flax (https://github.com/google/flax).
from flax.core.frozen_dict import FrozenDict
lfads_hps = FrozenDict(lfads_hps)

Thx! In the end I also use this same method to solve the problem =D


verify broken?

Hello, could it be the case that your verify message feature is broken? I am trying to verify the following msg that I believe was created using your sign message feature: {"address":"0xf73752c21404d457c502b23d3a81a1a179b86b3d","msg":"I, Tuan Le own this wallet. 0xf73752c21404d457c502b23d3a81a1a179b86b3d","sig": "0x4c515f099e93e3c90c5f40bc8237d961b2d89f6ce8a3afb54b7996d69ee8802b10b1b8fefe5c2d0045a7eea93e50f13d4145706ea5e2658a07d84a0aeaabbed21c","version":"2"}
2h 2 hours ago
etherchain.org
the console just prints the following error message: TypeError: Cannot read property 'getHWType' of undefined
at ChildScope.$scope.verifySignedMessage (etherwallet-master.js:2570)
at fn (eval at compile (etherwallet-master.js:44152), :4:177)
at callback (etherwallet-master.js:55774)
at ChildScope.$eval (etherwallet-master.js:46977)
at ChildScope.$apply (etherwallet-master.js:47077)
at HTMLAnchorElement. (etherwallet-master.js:55779)
at defaultHandlerWrapper (etherwallet-master.js:32707)
at HTMLAnchorElement.eventHandler (etherwallet-master.js:32695)

I’m on this atm, testing. I will come back with a pr

Will make a pr tomorrow.
0xf73752c21404d457c502b23d3a81a1a179b86b3d did sign the message I, Tuan Le own this wallet. 0xf73752c21404d457c502b23d3a81a1a179b86b3d.
Test it here: https://mytokenwallet.com/signmsg.html
Technical:
The problem was / is that the window has no scope to master, so we got an error, also if we did not unlock the / a wallet first we have no defined hwType, for that we got undefined.
Jo, one problem, but 4 files to change.
From user side:
Can not sign a message without unlocking any wallet before. So for a fast workaround just unlock any wallet and verify than.

I’m going crazy. Working and discovering since yesterday you opened this issue.
 We're getting closer but we are not there yet unfortunately. ☹

This may help...
            verifyMessage(signature, message) {
                return new Promise((resolve, reject) => {
                    if(signature && message){
                        const {v, r, s} = util.fromRpcSig(signature);
                        const message = util.toBuffer(util.sha3(message));
                        const publicKey  = util.ecrecover(message, v, r, s);
                        const addressBuffer = util.publicToAddress(publicKey);
                        const address = util.bufferToHex(addressBuffer);
                        resolve(address);
                    } else {
                        reject(Error('Missing Arguments'))
                    }
                });
            }```

will test it asap

If the hash signed contains the date, how we can get, after the transaction has been submitted, "the message before it was signed" parameter to pass to verifyMessage?

it still doesn't work.
{
  "address": "0x3444bc7b064c5fcd628bd91b5d505624a157e54f",
  "msg": "2",
  "sig": "0x967a96670b79422febb53d7ffdff7c56ae62fe20a46aca5c59748236f3940abe56e76a393756b127392972909e443f0e23d74c21c3fe7166269e9334730d33b51c",
  "version": "2"
}

I get the same error
 Cannot read property 'getHWType' of undefined


How to unable Enable bing spell checker in Luis?

Hi Folks,
I am facing the issue with the spell checker option in LUIS, As i am checked it and published. When i see again its showing as unchecked. I had purchased Bing spell check key too.
Please help me with this...
Thank you...!!

Hi @dileepkantapop, I'm not sure what Luis is. I would be happy to try to help. Do you have a link to project you are working with?

Luis is an Microsoft AI https://www.luis.ai/home
@dileepkantapop I think this thread will help you https://github.com/Microsoft/Cognitive-LUIS-Windows/issues/18#issuecomment-289385413

Hi @dileepkantapop , If you are referring to this screenshot, then the state of selection is not saved. It does not persists the selection.

However, the following are the steps,

Get a bing spell check API key
Add it in external Association
Publish it with Bing spell check API on
Every time the invocation is done from your code - have "spellCheck=true" is set.

https://westus.api.cognitive.microsoft.com/luis/v2.0/apps/c003afeb-e2b1-4890-a780-63a31812c735?subscription-key=8fb772cfd44c492a98280b53edb40c61&verbose=true&timezoneOffset=0&spellCheck=true&q=

buddy @yescay i am using the LUIS in Bot framework C# i can't call API to get the result. I know that it has "spellCheck=true" in API. But my problem is different! The state of the checkbox is not saving.

The state of the check box does not matter – the parameter value is what matters. Explore the Request object of your C# code to see if you can set the bing spellcheck that you can set to “true”.
AFAIK This checkox in LUIS publish page – does not retain state.
Thanks
Kamesh
From: Dileep Kanta Bivanapalli<EMAIL_ADDRESS>Reply-To: gunthercox/ChatterBot<EMAIL_ADDRESS>Date: Friday, 11 August 2017 at 5:42 PM
To: gunthercox/ChatterBot<EMAIL_ADDRESS>Cc: yescay<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [gunthercox/ChatterBot] How to unable Enable bing spell checker in Luis? (#922)
buddy @yescay i am using the LUIS in Bot framework C# i can't call API to get the result. I know that it has "spellCheck=true" in API. But my problem is different! The state of the checkbox is not saving.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Buddy! In Bot framework we won't call API, we just have Intents in which we will get data in entities.


[Feature] Add Endpoint to calculate Compound Annual Growth Rate (CAGR)

Are you adding a new feature?

[x] YES
[ ] NO

Are you enhancing the old feature?

[ ] YES
[x] NO

Is your feature request related to a problem?

[ ] YES
[x] NO

Describe your feature/enhancement.

The Compound Annual Growth Rate (CAGR) is a financial metric that measures the average annual growth rate of an investment over a specific period, accounting for compounding effects.
Formula:
CAGR = ((Final Value / Initial Value)^(1 / Number of Periods)) - 1
Parameters:
Initial Value: The starting value of the investment or asset.
Final Value: The ending value of the investment or asset after the specified period.
Number of Periods: The total number of periods (years, quarters, months, etc.) over which the growth occurred.

Can you add the feature by yourself?

[x] YES
[ ] NO

Additional Information

I think it is also a useful function to add, kindly review it.

Oh, my mistake.


Execution time reported for HTTP request by logger is lower than in reality

Describe the bug
I make an HTTP requests to REST API that returns quite a lot of data. I have now used HttpClientFactory from .Net Core to get the data and it works fine. But what has puzzled me is a discrepancy between my timing and internal timing that comes from the library itself.
Here is an example message from the library:
End processing HTTP request after<PHONE_NUMBER>ms - OK
The thing is: it really takes 20-30 more seconds to really download all the data even though the library has written it's all finished (To execute client.GetAsync() method).
I assume what happens here is that library is very eager to report right after the header is received without waiting for the body to get downloaded.
This really sound like a bug, but I am not sure if it is. Maybe someone has a better explanation on why this is happening?
To Reproduce
Steps to reproduce the behavior:

Using version '2.2.0.' of package 'Microsoft.Extensions.Http'
Run this code: Register HttpClient with HttpClientFactory to get logging, make a call that takes a lot of time and returns lots of data ('client.GetAsync()') and check how much time does it take
The time it takes is significantly higher than what the library itself reports in the log statement

Expected behavior
Library should log the end request when it is really finished, that is all the data is received.
Additional context
I am not 100% sure this is a bug, but would like at least to have som explanation on why this is happening then.
P.S. Here is an example of my code:
Stopwatch watch = new Stopwatch();
watch.Start();
var response = await Client.GetAsync(url);
watch.Stop();
Console.WriteLine($"Elapsed time {watch.ElapsedMilliseconds} ms");

Elapsed time in my log is way higher than in the system message
P.P.S I have tried running with HttpCompletionOption.ResponseContentRead but it really did not change the outcome, it's still a big difference.

That's interesting, do you have ability to profile this and see where the time is spent?

I can try, but I assume the reporting happens right after the header is received, and it takes some time to receive everything. But I don't really know, and probably profiling won't help either since it's not CPU usage

Most profilers have an option to profile 'thread time' instead of 'cpu time'. In particular I know that the timeline mode of JetBrains dotTrace supports this sort of thing, and they have a very generous trial period.
If you're to provide some data, or a simple repro I'd be happy to investigate this.
One other question - are you running in an environment with a sync context? Does it make a difference if you do await Client.GetAsync(url).ConfigureAwait(false)?
The log messages that you're reporting are from our outer logging handler, so I would expect them to be as accurate as possible.  Message handlers also act like middleware, and this one is the one that surrounds all of the pipeline. https://github.com/aspnet/Extensions/blob/master/src/HttpClientFactory/Http/src/Logging/LoggingScopeHttpMessageHandler.cs#L66
I would expect HttpCompletionOption.ResponseContentRead to have an effect on the timing of this, but I still don't have a guess as to what the cause of this might be.

FYI @karelz @stephentoub

OK I can definitely reproduce this with a "slow" server and HttpCompletionOption.ResponseContentRead
The reason why this is the case is that the support for HttpCompletionOption.ResponseContentRead is implemented inside HttpClient, not inside the message handler chain. You can see the code here
So when our logging code runs, it's always going to be just after the headers of the response have been read. I'm not sure that there's anything we can or should do about this - but there's definitely a reasonable explanation for it.
If for some reason you really want the body to be totally read during the message handler chain, you can write your own handler to force this earlier.
Example:
        public class MyBufferingHandler : DelegatingHandler
        {
            protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
            {
                var response = await base.SendAsync(request, cancellationToken).ConfigureAwait(false);
                await response.Content.LoadIntoBufferAsync();
                return response;
            }
        }

info: System.Net.Http.HttpClient.foo.LogicalHandler[100]
      Start processing HTTP request GET https://localhost:44316/Download?bytes=3145728&delay=1
info: System.Net.Http.HttpClient.foo.ClientHandler[100]
      Sending HTTP request GET https://localhost:44316/Download?bytes=3145728&delay=1
info: System.Net.Http.HttpClient.foo.ClientHandler[101]
      Received HTTP response after 732.418ms - OK
info: System.Net.Http.HttpClient.foo.LogicalHandler[101]
      End processing HTTP request after 12748.0145ms - OK
Elapsed time 00:00:12.7516826

Thanks for the explanation, that is pretty much what I expected to happen that logging code runs right after the headers are being read. It's not that critical for me to write a special handler, since I can rely on my logging that is made on top instead.
A little problem here is that it is a bit misleading to say "End processing HTTP request", since it is actually not quite "ended", depending of course on what we mean by that word.
So it does mean additionally if one wants to know how much time it takes end to end, it's better to have a logging on top yourself, rather than rely on internal logging that will always be off (with the delta depending on the size of the content, bandwidth, etc.)

I think you're right about that - do you have a suggestion about what this log message should say?

End receiving HTTP header or HTTP header received? Sounds a bit strange, but looks correct :)
I am a bit confused in your example about this log message that comes before end receive:
Received HTTP response that happens 10 seconds before the last message. If the last one is after headers are received, then when does the first message occur?

If the last one is after headers are received, then when does the first message occur?

In my example, I wrote a handler to force downloading of the request body. Since a call to HttpClient can trigger multiple actual requests, we have logging at both ends of the pipeline. In between is where the download of the message body occurs. It's academic though because we're not going to put this handler in the product code. I was just demonstrating that it's possible.

I hit this problem too.
When a call is made to GetAsync with content buffering option (which is the default if the completion option parameter is not specified), it is possible that the developer expects the operation to be very fast. If it turns out that it takes a significant time to complete on average, he might want to act on that by doing something after the headers have been received, before proceeding to load the content. In order to diagnose this need, he requires proper instrumentation.
There does seem to be a lack of opportunity to intercept the content buffering, specifically to log its duration separately from the duration of getting just the response headers.
The distinction between the 2 can give important clues. If you know that you have low network latency (thanks to ping), getting response headers late incriminates the server rather than the network. Getting the full content late might however incriminate the network bandwidth, unless network bandwidth tests say otherwise, in which case the server is streaming the response body from a slow source (obviously, as is too often the case, a 5.25" floppy disk).
Since the completion option is not passed to the message handlers, there's no way to know if the time of buffering should be taken into consideration.
Even if we had a way of getting the completion option within the pipeline, injecting a content buffering handler into the pipeline would unfortunately not allow the logging handler to discriminate the header response time from the content buffering time as it would only be able to measure the single SendAsync call to the inner handler.
Therefore, in that scenario, the logging handler would also have to be the one doing the content buffering. That sounds like a missed opportunity to separate responsibilities, which is a shame when using the chain of responsibilities pattern.
Perhaps the ideal solution is to allow interception of the content buffering in HttpClient separately from the chain of message handlers.
That can be done a number of ways:

Make SendAsync or some other method virtual to allow interception by overriding
Create an IHttpClient interface allowing for full customization, but that would impact the HttpClientFactory signature
Use a chain of responsibility pattern similarly to what is done for message handling, but this time for the content buffering (we'd then have a LoggingContentBufferingHandler calling the inner default ContentBufferingHandler), or in a more abstract way that resembles the parameter for the HTTP completion (e.g. HttpCompletionHandler)

There's a possible workaround that I'm going to investigate: adding a message handler that replaces the response content object (HttpContent) with a decorator on the original content object.
That decorator will override CreateContentReadStreamAsync by returning a stream that also decorates the stream of the original content object. The stream decorator will allow logging when the content reading starts, and when the end of the stream is reached, and of course get the duration between the 2.
It won't be meaningful if the stream is not read in one shot, but for my use cases, it nearly always is. A potential advantage is that it should work even if the content is not buffered and is read separately from the SendAsync call's completion.

I'm sending a small pr to adjust this logging message to be more clear. All of the real solutions to this problem seem like things that are either:

Out of scope for the client factory
OR have a big impact on how request processing works


table.group_by([]).aggregate raises an error on some aggregate functions.

Describe the bug, including details regarding any error messages, version, and platform.
Grouping on empty keys aggregates against the whole table (relates to #14896).  But there are 3 hash aggregate functions which do not have corresponding scalar aggregate functions: hash_distinct, hash_list, and hash_one. Grouping on empty keys with those raises a key error.
In []: table = pa.table({'key': list('aba'), 'value': [0, 1, 2]})

In []: table.group_by(['key']).aggregate([('value', 'list')])
Out[]:
pyarrow.Table
key: string
value_list: list<item: int64>
  child 0, item: int64
----
key: [["a","b"]]
value_list: [[[0,2],[1]]]

In []: table.group_by([]).aggregate([('value', 'list')])
...
ArrowKeyError: No function registered with name: list

In []: table.group_by([]).aggregate([('value', 'min')])
Out[]:
pyarrow.Table
value_min: int64
----
value_min: [[0]]

Component(s)
C++, Python

take


Feature request: allow to define destination variable name of parameter

Request: allow to define destination variable name of parameter (dest), like so:
@click.option('-or', dest='myname', help='What a relief...')
def mycommand(..., myname, ...)
    ...
if myname:
    ...
Alternatively, the Context could have a new parameter "name_prefix" or "name_suffix", such that "-or" translates to "myprefix_or" or "or_mysuffix".
argparse can do it
argparse allows to set the destination attribute name through the dest parameter which solves this problem: https://docs.python.org/3/library/argparse.html#dest
It would be great if Click had something similar.
Reasoning
I recently hit a problem which seems unsolvable with Click at the moment:
You cannot have option/argument names which resemble Python keywords:
@click.option('-or', help='This will not work')
The resulting variable name is "or" which is a python keyword and thus cannot be used:
@click.command()
def mycommand(..., or, ...)
SyntaxError: invalid syntax
The same goes for things like "-if" and other Python keywords. "-if" would, for example, be a great shortcut for something like "input files", so it hurts that you cannot use it without a long name.
Sure, you could cheat and add a long name - but what if you don't want to, e.g. because you want to keep the lest column in the help text slim to leave more space for text on the right?

Sorry, please close. Click already supports this, I just could not find it in the documentation:
@click.option('-or','myname',help="Works")
It is hidden in this section of the Option documentation : https://click.palletsprojects.com/en/7.x/options/#feature-switches ("transformation")
It would be great if the documentation mentioned it more succinctly.


Error selecting GPU programmatically from jupyter

I have a multi-GPU machine and am trying to train different models on different GPUs. I'm trying to set the GPU programmatically rather than use env vars, but am running into issues. Here's my code:
def choose_gpu(gpu_id):
    os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
    with K.tf.device(f'/gpu:{gpu_id}'):
        config = tf.ConfigProto(intra_op_parallelism_threads=10,
                                inter_op_parallelism_threads=10,
                                allow_soft_placement=True,
                                device_count={'CPU': 1, 'GPU': 1})
        session = tf.Session(config=config)
        K.set_session(session)

However, when I try to run the code in two separate jupyter notebooks, using ids 0 and 1, I get the following error:
2019-03-04 10:36:27.210489: W tensorflow/compiler/xla/service/platform_util.cc:240] unable to create StreamExecutor for CUDA:0: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY: out of memory; total memory reported:<PHONE_NUMBER>6
2019-03-04 10:36:27.210803: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x56311c4bccf0 executing computations on platform CUDA. Devices:
2019-03-04 10:36:27.211230: F tensorflow/stream_executor/lib/statusor.cc:34] Attempting to fetch value
instead of handling error Invalid argument: device CUDA:0 not supported by XLA service

It works fine if I set CUDA_VISIBLE_DEVICES at the top of the script, but I'd rather not have to manage this. Any sense of what I may be doing wrong? I'm using

keras 2.2.4
tensorflow 1.13.1
ubuntu 18.04
jupyter 5.2.4
jupyter lab 0.35.4

environment param need to be set before you import keras

Even when I do that, one of the instances crashes at the end of the first epoch. I don't even get any error message. Just a message that the kernel was reset.

@lminer  Is this still an issue for you? I have cross posted this issue on TF repo  as well for TF folks to take a look. Here is the link to it. Thanks!

@ymodak I think I solved the problem. Kernel reset was a different issue.

Cool. I will go ahead and close this issue now. Thanks!

@lminer and @ymodak what was the solution to this problem ?

@lminer How did you manage to solve this issue?

I have four GPUs on my machine (Ubuntu 1804).
Setting:
export CUDA_VISIBLE_DEVICES=0,1,2,3
in the shell, or
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2,3'

worked for me.


HTTP/2 in Aerys not working

Hi,
I just tried to make a HTTP/2 request to the demo page of aerys, but I get a StackTrace once the request is sent.
See screenshot for details.

This is the full error message:
[07:39:09] error Error: Class 'Aerys\Failure' not found in /root/fuzzing/server/aerys/lib/Http2Driver.php:275
Stack trace:
#0 [internal function]: Aerys\Http2Driver->writer(Object(Aerys\InternalRequest))
#1 /root/fuzzing/server/aerys/lib/functions.php(476): Generator->send(NULL)                                             [0/354]
#2 [internal function]: Aerys\responseCodec(Object(Generator), Object(Aerys\InternalRequest))
#3 /root/fuzzing/server/aerys/lib/StandardResponse.php(257): Generator->send(NULL)
#4 /root/fuzzing/server/aerys/lib/StandardResponse.php(162): Aerys\StandardResponse->end('<html><body><h1...')
#5 /root/fuzzing/server/aerys/demo.php(4) : eval()'d code(17): Aerys\StandardResponse->send('<html><body><h1...')
#6 /root/fuzzing/server/aerys/lib/Router.php(68): Aerys\Bootstrapper->{closure}(Object(Aerys\StandardRequest), Object(Aerys\Sta
ndardResponse), Array)
#7 /root/fuzzing/server/aerys/lib/Bootstrapper.php(179): Aerys\Router->__invoke(Object(Aerys\StandardRequest), Object(Aerys\Sta
ndardResponse))
#8 [internal function]: class@anonymous->__invoke(Object(Aerys\StandardRequest), Object(Aerys\StandardResponse))
#9 /root/fuzzing/server/aerys/vendor/amphp/amp/lib/functions.php(775): Generator->current()
#10 /root/fuzzing/server/aerys/vendor/amphp/amp/lib/functions.php(763): Amp\__coroutineAdvance(Object(Amp\CoroutineState))
#11 /root/fuzzing/server/aerys/lib/Server.php(749): Amp\resolve(Object(Generator))
#12 /root/fuzzing/server/aerys/lib/Server.php(701): Aerys\Server->tryApplication(Object(Aerys\InternalRequest), Object(class@an
onymous))
#13 /root/fuzzing/server/aerys/lib/Server.php(561): Aerys\Server->respond(Object(Aerys\InternalRequest))
#14 /root/fuzzing/server/aerys/lib/Server.php(539): Aerys\Server->onParsedMessageWithoutEntity(Object(Aerys\Client), Array)
#15 /root/fuzzing/server/aerys/lib/Http1Driver.php(302): Aerys\Server->onParseEmit(Array, Object(Aerys\Client))
#16 [internal function]: Aerys\Http1Driver->parser(Object(Aerys\Client))
#17 /root/fuzzing/server/aerys/lib/Server.php(532): Generator->send('GET / HTTP/1.1\r...')
#18 /root/fuzzing/server/aerys/vendor/amphp/amp/lib/NativeReactor.php(257): Aerys\Server->onReadable('000000005be04d7...', Reso
urce id #131, Object(Aerys\Client))
#19 /root/fuzzing/server/aerys/vendor/amphp/amp/lib/NativeReactor.php(244): Amp\NativeReactor->doIoCallback('000000005be04d7...
', Object(stdClass), Resource id #131)
#20 /root/fuzzing/server/aerys/vendor/amphp/amp/lib/NativeReactor.php(214): Amp\NativeReactor->selectActionableStreams(0.3786)
#21 /root/fuzzing/server/aerys/vendor/amphp/amp/lib/NativeReactor.php(70): Amp\NativeReactor->doTick(false)
#22 /root/fuzzing/server/aerys/vendor/amphp/amp/lib/functions.php(46): Amp\NativeReactor->run(Object(Closure))
#23 /root/fuzzing/server/aerys/bin/aerys(96): Amp\run(Object(Closure))
#24 {main}

closed by 7740b7f  could you re-try with latest master?

That's not the root cause though (there should be no Failure instantiated at all)

The issue was the RFC requiring "URL-safe" base64 … added a strtr, fine now.

See comment on last commit.

@kelunik I see no comment on the latest commit?

https://github.com/amphp/aerys/commit/52e51e10c46f1fd73141fb8a916da401c7e3c112#commitcomment-16029591

Thanks for quick fix. It's working now.

Thanks for the report and confirmation. :+1:


Problem with iOS and MacOS clients v1.2.1(Oct 19 2018 release)

I've been using Outline on iOS for many months now (with a Digital Ocean droplet).  Many thanks!
Ever since the Oct 19 update to v1.2.1, however, I have been having problems.
Specifically: periodically, the iOS Outline client (on iPhone 8+, iOS 12.0.1/16A404) gets into the following state:

No traffic seems to get through (e.g., trying to bring up any arbitrary web page -- even something well-known, such as cnn.com or google.com -- spins forever, and Safari eventually gives up)
The "VPN" indicator is still lit in the top left of my iOS status bar
If I open the Outline client, it's in a "reconnecting" loop.  It seems to keep trying forever (or at least I haven't been patient enough to wait for it to give up -- I've always tapped "disconnect" to make it stop after 10-20 tries).

After Outline disconnects, the "VPN" indicator (correctly) goes out, and traffic starts flowing again.  If I immediately try to make the Outline client connect again, it tries several times and eventually gives up.
This only seems to happen when on wifi.  I have noticed that this seems to happen when I move from cellular to wifi, but I don't know if this is always the case.
HOWEVER, when this problem occurs, I notice that if I leave Outline disconnected for a few minutes and then go back into the Outline client app and try to connect again, it connects just fine (i.e., the "VPN" indicator goes on and traffic is still flowing).
Did something change in the v1.2.1 release with regards to cached connections to my Outline server, perchance?  Or did something regarding my connection to my Outline server become sensitive to changing network routes (e.g., switching from cellular to wifi)?
I ask because I see "Bypass LAN and private networks in the VPN" in the v1.2.1 release notes.  What exactly does that mean?
Please let me know if there's more information that I can provide to help debug this issue.  Thanks!

One clarification: I noticed that even after some time on the same wifi network, the iOS Outline client can get in the same failed state (i.e., no traffic gets through, the Outline client gets stuck in a "reconnecting" loop).
Specifically:

Last night when I filed this github issue (around 10pm), I had my iPhone successfully connected to the Outline server, and traffic was flowing fine.  I went to sleep shortly thereafter.
I see in my iCloud logs that my phone successfully performed an iCloud backup at 2:04am last night.
But when I got up around 6:30am this morning, Outline was in the failed state (i.e., no traffic could get through -- e.g., I couldn't load any web pages -- and the iOS Outline client was stuck in the "reconnecting" loop).
I tapped "disconnect" in the Outline client and traffic started flowing again.
About 45 minutes later (without having left the same wifi network), I tapped "connect" in the Outline client, it successfully connected, and all traffic continued to flow.

Thanks for the detailed report, @jsquyres. I have been investigating this issue and, unfortunately, have found it hard to reproduce consistently. In my testing, I have seen the connection fail with a particular server and network combination. However, it is strange that the same conditions work fine with the macOS client, which shares the code base with the iOS client. Furthermore, a different iOS device will also connect to the same server over the same network.
I believe that this is not related to the latest update, which introduced LAN and private network bypass (meaning that traffic to your local network does not go through the VPN), as I disabled this feature while testing, yet the problem persisted. My suspicion is that the app is not handling network changes correctly; I'll keep looking and update this thread with any information.
The solution that has worked for me is to reboot the phone. Can you please give that a try and let me know if that helps? It would also be useful if you could submit feedback through the app and tag it with this GitHub issue ID.

I can confirm that the problem occurs on both macOS and iOS devices (versions 10.14 and 12.0.1, respectively). Switching between Wi-Fi and cellular, or even sending computer to sleep and waking it up in a couple of minutes, often leads to VPN connection issues. The symptoms are just like @jsquyres has described: the traffic starts going through as soon as VPN is disabled, and attempts to enable VPN result in some waiting and eventual failure.
I tried setting up a brand-new server (it was more than just one server—it was about a dozen), but this doesn’t seem to be helpful.
On macOS, the only working solution I’ve been able to find so far is opening System Preferences → Network → Advanced, switching to the TCP/IP tab, setting IPv4 to Off, saving and applying the configuration, then setting IPv4 back to Using DHCP.

@alalamav It happened again this morning exactly the way it happened yesterday: I went to bed last night with my iPhone on my home wifi, the Outline client was active / "VPN" icon was displayed in the top left -- traffic was flowing, and everything was fine.  I see that my iPhone did an iCloud backup last night at 11:31pm.  This morning I woke up around 6:15am, and the iOS client is in the failed state again (no traffic is flowing, the Outline client is in the eternal trying-to-connect state).  I tried to submit feedback from the app, but of course it failed to submit, so I backed out of feedback, tapped "disconnect" to make the the client disconnect, and then I was able to submit the feedback properly through the app (per request, I cited jigsaw-code/outline-client #352 / @jsquyres).  I don't know if submitting feedback while the client is disconnected changes the state data that the app submits with the report.
@yakovmanshin Thanks for the report.  I renamed the issue to include "MacOS", too.

I forgot to mention 2 things:

I rebooted my phone yesterday morning, per @alalamav's suggestion.  This is the first time I've seen the problem since rebooting my phone.
After the failed state this morning:

I failed to remember to try to tap "Connect" again immediately after the failed state (I blame the lack of caffeine; per above, it is before 7am in the morning).  Oops.  If it behaves like it did before, it would have gone into the loop of trying-to-connect, but I didn't verify that this morning.  ☹
That being said, after the Outline client was disconnected for ~5 minutes, I did remember to tap "Connect" again, and it connected successfully / traffic is flowing / everything is fine.

If there's any kind of debug mode that I can activate in the app to gather more data when it's in this state, I'm happy to activate it.  Or if getting network traces of what traffic the phone is sending while in the failed state would be helpful, ...etc., let me know.  I'm a developer myself; iOS / mobile apps aren't my specialty, but I'm not afraid to get my hands dirty with deeply technical things if it would help get you more information.

Update: I caused the problem to happen again this morning by leaving my wifi, going on cellular for a while, and then joining a startbucks wifi.  My iPhone went into the failed state.
This time, I did remember to try having the Outline client connect immediately after disconnecting.  🎉
It failed, but in a slightly different way.  It only tried to connect a small number of times before giving up (note that other traffic is flowing just fine when this happens).  Specifically, it showed the circles-growing graphic 2-4 times before giving up and displaying the following (I obscured my server IP address):

This is a slightly different failure mode, which seems to indicate a different code path.  Perhaps that is helpful in tracking down the issue...?
Note that after waiting 5 minutes, I tapped "Connect" again and it connected just fine / traffic is flowing / etc.

This morning, I checked what network traffic my phone is sending/receiving while it is in the failed state.
tl;dr
When my phone is in the failed state, it doesn't actually appear to be sending any traffic to my Outline server, even though it claims to be trying to reconnect.
More detail
Interestingly, when my phone is in the failed state, I only see 2 things:

My phone periodically ARPs for the gateway
My phone is sending isakmp-nat-keep-alive packets to an IP address that is not my Outline server

In the tcpdump output below, my phone is <IP_ADDRESS>, aka 70:ef:00:34:98:11, and the gateway is <IP_ADDRESS>, aka 00:08:a1:24:03:a2.  All the traffic occurs on VLAN 10:
11:38:29.428381 70:ef:00:34:98:11 > 00:08:a1:24:03:a2, ethertype 802.1Q (0x8100), length 47: vlan 10, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 31155, offset 0, flags [none], proto UDP (17), length 29)
    <IP_ADDRESS>.4500 > 129.192.x.y.4500: [no cksum] isakmp-nat-keep-alive
	0x0000:  4500 001d 79b3 0000 4011 0e88 c0a8 0a22<EMAIL_ADDRESS>

It seems to be isakmp-nat-keep-alive'ing with 129.192.x.y (I removed the last 2 octets), and I don't know who that is.  I do have my employer's MDM on my phone, though -- I don't know if this could be related.
This is the only traffic that my phone is sending while the phone is in the failed state.  I'm fairly un-knowledgeable about how VPNs / encrypted proxies work, but this feels significant.  I.e., it doesn't appear to actually be sending any traffic to my Outline server / trying to reconnect.
Once I tap "disconnect" in the Outline client, however, I get a torrent of traffic to/from a bunch of different peers (as expected).
After I am able to successfully reconnect to my Outline server, then -- also as expected -- nearly all the traffic from my phone is going to my Outline server.  The only two exceptions (from a 60 second tcpdump slice) are:

DNS and MDNS queries, both of which go to the <IP_ADDRESS> gateway, cited above.
The same isakmp-nat-keep-alive's to the same 192.192.x.y host, which isn't on my local network.

I can confirm, this is an issue. Happens a lot after wake up from sleep.

Thank you for the detailed reports, @jsquyres, and thanks for @philkunz and @yakovmanshin for flagging the problem on macOS. These are very useful.
I found this post that describes behavior consistent with this issue and suggests there may be a bug in the OS when applying routing rules on connectivity changes. Another hypothesis is that we are excluding the virtual address that the VPN binds to (<IP_ADDRESS>); thus Outline stops receiving traffic, as @jsquyres indicated.
We have decided to roll back the LAN bypass feature for iOS and macOS until we understand how it is affecting the observed behavior. As we start to think of ways to fix this issue without affecting all users, would any of you be willing to join our beta channel through TestFlight? If so, please send an email to<EMAIL_ADDRESS>with your Apple ID (email address) and subject line "Test Flight". Note that TestFlight is only available for iOS apps.
Thank you for the help!

Glad to help.  I sent mail to<EMAIL_ADDRESS>I'm happy to be part of the Test Flight.

Thank you @jsquyres. The email address is actually<EMAIL_ADDRESS>Would you mid re-sending your information? I apologize for the mistake.
The iOS client v1.2.2 has been released in the App Store. Please reopen this issue if the problem persists.

Just to followup: after updating to iOS 1.2.2, the problem has disappeared.  I've had no connectivity problems since updating.

It was my search to find a solution to the constant on and off connection that brought me to this post.
The problem described here is not limited to IOS. I am experiencing the same problem on the ubuntu server on a Windows Hyper-V.
I have created two servers now and both are losing connection intermittently.
Please, what solution do you suggest?
Thank you


Change App Language in settings

Describe your suggested feature
Since some time ago, I noticed most of the app's text is localized to my device's language. Nice localization work!
But I prefer to use TachiyomiSY with English texts, and I didn't find a setting to change the app's language.
Can this language selection be added to the settings? Or does this feature already exist and I just didn't find it?
Other details
I'm using Preview r407.
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
[X] I have written a short but informative title.
[X] If this is an issue with an extension, I should be opening an issue in the extensions repository.
[X] I have updated the app to version 1.7.0.
[X] I will fill out all of the requested information in this form.

Taken from the Tachi discord.
Changing the app language has been removed due to it causing small bugs everywhere in Tachiyomi.
Android does not provide a standard way of selecting a language that's not your device-language so it was a messy implementation.


Build broken with v2 release

The master build is currently broken following the recent v2 modifications.
https://travis-ci.org/braveg1rl/performance-now/builds/189923759
It looks like these are just tweaks to the acceptance criteria (loosening the expectation), but I want to be sure before upgrading.

I think calling the build "broken" is a bit too much. One or more of the tests on all but one of the various test runs Travis CI servers fail. One of the test runs succeed.
It's because the values reported by the scripts are highly variable.
I think the failing tests only reflect some unwise choices on my part on how to assure myself I was writing the good code. I wrote the tests for the very purpose of moving safely from v1.0 to v2.0. First, I made tests that passed for the v1.0 logic, then saw them fail for the 2.0 logic (as expected), and then adjusted them to make them pass again.
Based on my extensive testing, I think the logic of v2.0 is absolutely sound.
I very much need to reconsider how I do the testing, though.

Excellent, your conclusion is in-line with mine.  I appreciate your feedback!


Can't handle arrays

this doesn't work, while it should:
import { html } from 'https://esm.sh/rehtm'

document.body.appendChild(html`
  <ul>${[1, 2, 3, 4, 5].map(i => html`<li>${i}</li>`)}</ul>
`)

this is working now.


Added test cases for Format, ProgressBar, Reader pkgs.

What type of PR is this?
/kind feature
What this PR does / why we need it:
This PR adds test cases for the formatInfo, NewTransport, and NewReader functions in the pkg/utils/progressbar package.
Does this PR introduce a user-facing change?
NONE

Hey @wzshiming I've added the test files for the format, progressbar, reader.go files. Can you please have a look at them?

@wzshiming I've made the required changes and also squashed all the commits. Can you take a look at them?


Added inc dec stock feature in user guide

Make the Feature more intuitive as well -> supplement with screenshots.

Codecov Report

Merging #169 (eaa23df) into master (617a4e2) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #169   +/-   ##
=========================================
  Coverage     68.77%   68.77%
  Complexity      708      708
=========================================
  Files           112      112
  Lines          2488     2488
  Branches        302      302
=========================================
  Hits           1711     1711
  Misses          667      667
  Partials        110      110

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Decide support status of K3s

We have had a number of issues reported for K3s that we don't have devs with the resources or setups to properly investigate. We should decide if we really can support K3s, and if not remove it from our docs to not confuse users.

Related issue: https://github.com/submariner-io/subctl/issues/444

Per discussions with the team, I think we don't have folks with the infra or cycles to support K3s currently. We'd very much like someone from Rancher or who otherwise wants K3s support to adopt that part of the project.
We've already remove K3s docs from the website to not mislead folks.


chore(main): release 2.6.2

:robot: I have created a release beep boop
2.6.2 (2022-02-03)
Bug Fixes

docs/application: Fix formatting in "Using Traits in Verticles" tutorial (58b44ff)
docs/general: :adhesive_bandage: Fix more language inconsistencies (ed91fa8)
docs/general: :adhesive_bandage: Fix various language inconsistencies (4dec5a0)
docs/internal: :bug: Fix typo in Error Messages Concept (eb93e41)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/wuespace/telestion-docs/releases/tag/v2.6.2 :sunflower:


Attempt to fix #2: Hard parser crashes

I replaced all uses of unwrap() in the parser and replaced them with error handling where possible.
Unfortunately, that required to rewrite some nice iterators to loops.
Switched implementation of device discovery to use tokio sockets, which allowed to remove the sleep(500ms) hack, some unsafe code, and the socket2 crate dependency. This further also seems to fix some problems with indefinite hangs I had - probably due to the mixture of tokio async and blocking code.
Silently ignore parsing errors in device discovery stream, don't crash

Warning: Changed discover_pnp_locations() signature
I ran rustfmt on the commit, I hope that was ok!
Hopefully fixes #2

I don't know if that fixes everything yet, because the problematic device isn't responding anymore - was probably switched off (wasn't mine).

Great work on the pull request! It's always good to see code improvements that remove unnecessary unsafe code and make the code more reliable.
Regarding the changes you made, it's great that you replaced the uses of unwrap() with error handling where possible.
However, as you've mentioned, you have made some breaking changes to the discover_pnp_locations() function which has resulted in a change in its signature.
As a result, I would suggest updating the examples in the README file to reflect these changes. This will help new users understand how to use the updated function correctly.
Overall, the changes look good and should help improve the reliability and maintainability of the code. Thank you for your contributions!

Yes! Please give me some more minutes, found some more I could clean up

Okay, I think now I'm done!


Import models that are imported inline by odmlib to support compiled …

…define validation
Steps to test

Compile core using the pyinstaller command found in the README
Run a validation with directory including a define 2.1 file: ex ./dist/core/core validate -s sdtmig -v 3-4 -d msg/
Verify the validation runs completely and an excel file is generated

I've also created an issue here: https://github.com/swhume/odmlib/issues/14


[ER]: Set ETA dates based on Prod Rel

Description: (details of what and how should work)
Dev ETA and Test ETA need to be filled in all Features that have Production Release set. Therefore there may be a helpful functionality to automatically set them with a trigger of on ProdRel change.
Additional Context & Ad Hoc Info:
POs to be asked for details

Following may be a good entrypoint, or at least helper: #49


Docker improvements

@pslobo What do you think?

Looks good to me. I'll add my changes once you've merged those too.

@pslobo actually i need to test this more so instead of you creating a pull request that merges to master, you can make it merge here. we only have to make sure we merge this PR first.


Review and enhance metric support for product catalog service (Go)

The following are requirements for metrics in the project. Not all services will meet all requirements.

[ ] Service metrics should not be duplicative of trace-derived metrics.
[ ] Services should emit multiple business metrics where appropriate.
[ ] Services should emit trace exemplars for their metrics.
[ ] Services should use push metrics where possible.
[ ] Services should explicitly read baggage where needed.
[ ] Collectors should perform metric transforms to normalize resource and service metrics.

Referencing issue: https://github.com/open-telemetry/opentelemetry-demo-webstore/issues/43

@cartersocha I'm a little confuse, what metrics should we add to service? in my view, we can add some span metric by the otel-collector:

calls_total
latency_bucket

We haven’t had time to define our metric story yet. We are planning on discussing overall metric requirements in our next sig meeting.
We’ll need to set up a consumption experience too. I’m guessing we’ll add grafana in a container or something similar.
Feel free to open a pr with what metrics you think would be useful. Then we can use that to solicit feedback / input from everyone

The newest implementation instrumentation for the runtime package is in process, waiting for its finish.

Basic metrics have been added; closing issue.


Change Special from green to red

Hi there! Thanks for a really really nice color scheme. Love it :heart:
Most of my files are JavaScript and Markdown and I use pangloss/vim-javascript. I noticed that with ES6, some special characters in regular expressions and especially template strings are not highlighted. I found that this is because the "Special" color has the same value as the one for "String". I played around a little and found red to be a nice touch.
On this screenshot, left are two examples of the current version, and to the right with this change:

What do you think?

Hi,
It looks great, thanks!


PrivateRoute params support

Hi Tim, while working with react-devise, I was trying to use a param in  but the child component was not able to receive the param.  Example:
<PrivateRoute path="/welcome/:number" layout={MainLayout} component={Welcome} />
By changing the above to: <Route path="/welcome/:number" layout={MainLayout} component={Welcome} />
The child component started to receive the param as expected. Was this intentional with
<PrivateRoute>? Is it possible to have <PrivateRoute> support url params? Thanks!

I don't understand the question. It looks like you tried to provide an example, but didn't. This module should not affect access to URL params.

sorry about that, updated the comment with the correct code formatting so the example appears.

Fixed in v0.0.8.


Delay speed fix

The provided frame delay is never set on the created frame so the default is always used. This sets the delay in units of 10ms according to https://docs.rs/gif/latest/gif/struct.Frame.html#structfield.delay

Makes sense to me, thanks for the PR!


Document not-supporting nested configs and reasoning

We don't support nested configs and probably(?) won't.
This is because when we lint a file we only get one of the files rather than a list of all files. Potentially we could figure out a way to scan the tree for swiftlint configs and and pass them all through and handle it properly but that would be pretty involved and involve some changes to hound proper to support it.
This should be documented in the readme and probably on https://houndci.com/configuration as well.

Is there an example repo that uses nested configs?

No, I have yet to see anyone use it but noticed it's a feature swiftlint supports.

Both JSHint, and ESLint has something similar aswell.
So documenting it seems like a good idea.


feat:added password reset feature and using nodemailer to request password reset

It was an amazing experience working with Zod for configuration. I had already practiced the reset feature, so I revisited it by contributing. I also used Nodemailer to confirm the reset password.

@Pavankumar07s this PR incude lots of unusual changes and lots of file is getting delete can you please create PR for only those file that have changed ?

Alright 👍


error while installing folly dependency on Ubuntu 14.04 EC2 Server

I've been trying to install the dependencies for fbcunn but am running into the following error below - I'm thinking it may be a compiler issue but I just opened up this ec2 instance and am not completely familiar with the system yet. Any ideas?
make[1]: Entering directory `/tmp/fblualib-build.c898EK/folly/folly'
Making all in .
make[2]: Entering directory `/tmp/fblualib-build.c898EK/folly/folly'
depbase=`echo Bits.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
	/bin/bash ./libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H   -I./.. -pthread -I/usr/include   -std=gnu++0x -g -O2 -MT Bits.lo -MD -MP -MF $depbase.Tpo -c -o Bits.lo Bits.cpp &&\
	mv -f $depbase.Tpo $depbase.Plo
libtool: compile:  g++ -DHAVE_CONFIG_H -I./.. -pthread -I/usr/include -std=gnu++0x -g -O2 -MT Bits.lo -MD -MP -MF .deps/Bits.Tpo -c Bits.cpp  -fPIC -DPIC -o .libs/Bits.o
In file included from Bits.cpp:19:0:
./../folly/CpuId.h:160:19: sorry, unimplemented: non-static data member initializers
./../folly/CpuId.h:160:19: error: ISO C++ forbids in-class initialization of non-const static member 'f1c_'
./../folly/CpuId.h:161:19: sorry, unimplemented: non-static data member initializers
./../folly/CpuId.h:161:19: error: ISO C++ forbids in-class initialization of non-const static member 'f1d_'
./../folly/CpuId.h:162:19: sorry, unimplemented: non-static data member initializers
./../folly/CpuId.h:162:19: error: ISO C++ forbids in-class initialization of non-const static member 'f7b_'
./../folly/CpuId.h:163:19: sorry, unimplemented: non-static data member initializers
./../folly/CpuId.h:163:19: error: ISO C++ forbids in-class initialization of non-const static member 'f7c_'
make[2]: *** [Bits.lo] Error 1
make[2]: Leaving directory `/tmp/fblualib-build.c898EK/folly/folly'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tmp/fblualib-build.c898EK/folly/folly'
make: *** [all] Error 2

sorry about this. I just fixed the fblualib install script so that this error does not happen now.
I made sure tagged versions of folly and fbthrift are installed, rather than trunk. I tested the install process on a fresh ubuntu 14.04 machine.
You'll have to redo the install process starting from this step:
https://github.com/facebook/fbcunn/blob/master/INSTALL.md#install-folly-fbthrift-thpp-and-fblualib
If you are having installation errors after retrying the install once more (after my fresh fixes),
I can post the binaries here for Ubuntu 14.04 (they will be specific to ubuntu 14.04).

I have the same issue.
libtool: compile:  g++ -DHAVE_CONFIG_H -I./.. -pthread -I/usr/include -std=gnu++0x -g -O2 -MT Bits.lo -MD -MP -MF .deps/Bits.Tpo -c Bits.cpp  -fPIC -DPIC -o .libs/Bits.o
In file included from Bits.cpp:19:0:
./../folly/CpuId.h:160:19: sorry, unimplemented: non-static data member initializers
./../folly/CpuId.h:160:19: error: ISO C++ forbids in-class initialization of non-const static member 'f1c_'
./../folly/CpuId.h:161:19: sorry, unimplemented: non-static data member initializers
./../folly/CpuId.h:161:19: error: ISO C++ forbids in-class initialization of non-const static member 'f1d_'
./../folly/CpuId.h:162:19: sorry, unimplemented: non-static data member initializers
./../folly/CpuId.h:162:19: error: ISO C++ forbids in-class initialization of non-const static member 'f7b_'
./../folly/CpuId.h:163:19: sorry, unimplemented: non-static data member initializers
./../folly/CpuId.h:163:19: error: ISO C++ forbids in-class initialization of non-const static member 'f7c_'
make[2]: *** [Bits.lo] Error 1
make[2]: Leaving directory /tmp/fblualib-build.JycSeo/folly/folly' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory /tmp/fblualib-build.JycSeo/folly/folly'
make: *** [all] Error 2


METRON-990: Clean up and organize flux properties

Contributor Comments
This PR is mainly a refactor of the enrichment and indexing flux files along with their matching property files.  The changes include:

moving important settings with hardcoded values in flux files to property files so that they are configurable through Ambari
removing old and unused properties
elasticsearch.properties file is now implemented as a jinja2 template in the mpack, matching the enrichment.properties implementation
global.json is now implemented as a jinja2 template in the mpack
properties are now organized in Ambari as separate tabs and sub sections, hopefully making them easier to find
changed a couple properties to use a dropdown widget in Ambari

I wrote descriptions for new properties in a similar style as existing properties.  I feel the descriptions are a little short, curious if others agree.  I also stopped short of improving ALL our properties with better widgets than just a text box.  I imagine people will have opinions on how to best present properties in Ambari but this is a start.
This has been tested on full dev with the usual process.  When reviewing, spin up full dev and navigate to the Metron service in Ambari.  The Config section should look as described in the section above.  The enrichment.properties and elasticsearch.properties should look much shorter and easier to read.
One more thing.  I added a rat exception for *.json.j2 since we already have an exception for *.json (no comments in json is the reason?).  Let me know if that's wrong.
Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:

[x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
[x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

[x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?

[x] Have you included steps or a guide to how the change may be verified and tested manually?

[x] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via:
mvn -q clean integration-test install && build_utils/verify_licenses.sh

[x] Have you written or updated unit tests and or integration tests to verify your changes?

[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

[x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:
cd site-book
mvn site

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

Still working on updating the integration tests with the new properties.  Will have them fixed shortly.

Your updates look good @merrimanr. Just running it up again for a round of testing.

+1  I was able to alter settings for the Enrichment topology, successfully restart all services, and see the new configuration changes take effect.
Splitting out all of the settings across different tabs is a really nice way to do it.  Would have been a mess if you hadn't gone that extra mile.  Nice


Distinction between Opera and Opera Mini

Added distinction between Opera  and Opera Mini. Even if Opera Mini is an Opera's browser it stays very different from other Opera browsers because it's a proxy browser and it has its own behaviors when rendering web pages or while processing Javascript.
This change will allow to target specifically Opera Mini on mobile without affecting the regular Opera browser.
P.S: Now when you download Opera Mini on a mobile, you can switch between Opera  and Opera Mini by choosing the data saving mode, so you have one app which could be Opera or Opera Mini according to your settings.The User-Agent changes too as you are using the app as Opera or as Opera Mini
Referring to the issue #70
More about the difference between Opera and Opera Mini here

Mobiledetect also groups all Opera browsers, not sure if we want to divert too much from their detection in this case.

Ofcourse Mobiledetect groups them too but I sincerely believe that it's a mistake.
Opera's browsers can all be grouped but without "Opera-Mini".
To figure it out, you can download "Opera-Mini" on android or iOS, select the data save mode to really use the App as "Opera-Mini" and give it a try.  You will certainly notice the big difference compared to other Opera's browsers

@jenssegers @evrard-c I believe this would be an extremly useful feature.
Opera Mini users generally use opera's proxy and tend to handle requests in a drastically different way to other browsers, so being able to detect them would be extremely beneficial.


Implement Form Validation behavior for all our form components

Depends on

Should be done after #570

Why?
All form components beside the input should also be able to show an invalid state to inform the user that the entered value is invalid and he has to perform further actions.
Design

Figma - textarea
Figma - select

Acceptance criteria

Implement translated error message like in OnyxInput for all other existing form components that we have.

[ ] OnyxTextarea
[ ] OnyxSelect

[ ] In case of an error, a short text summary is displayed below the input with a more detailed text in the tooltip.
[ ] a custom error message can be passend
[ ] inside the alpha-test-app, inside the FormDemoView, include all modified form components from this ticket as examples

Definition of Done

[ ] example usage added to alpha-test-app
[ ] covered by component tests
[ ] screenshot tests are updated
[ ] follow-up tickets were created if necessary
[ ] updated version + documentation is deployed
[ ] Storybook can show the feature
[ ] Set task to In Approval

Review

add link to deployed storybook
[ ] approved by designer @jannick-ux
[ ] approved by dev add dev tag

Implementation details

use :invalid pseudo CSS-selector

closed, contents was transferred to
#570


'Message' is not JSON serializable

trying to run the first example but got the following error.
the code is cloned from the repository.
Traceback (most recent call last):   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1997, in __call__     return self.wsgi_app(environ, start_response)   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1985, in wsgi_app     response = self.handle_exception(e)   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1540, in handle_exception     reraise(exc_type, exc_value, tb)   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise     raise value   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app     response = self.full_dispatch_request()   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request     rv = self.handle_user_exception(e)   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception     reraise(exc_type, exc_value, tb)   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise     raise value   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request     rv = self.dispatch_request()   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request     return self.view_functions[rule.endpoint](**req.view_args)   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/flask-celery-example/app.py", line 75, in index     send_async_email.delay(msg)   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/celery/app/task.py", line 413, in delay     return self.apply_async(args, kwargs)   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/celery/app/task.py", line 536, in apply_async     **options   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/celery/app/base.py", line 737, in send_task     amqp.send_task_message(P, name, message, **options)   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/celery/app/amqp.py", line 554, in send_task_message     **properties   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/messaging.py", line 169, in publish     compression, headers)   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/messaging.py", line 252, in _prepare     body) = dumps(body, serializer=serializer)   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps     payload = encoder(data)   File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py", line 99, in __exit__     self.gen.throw(type, value, traceback)   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/serialization.py", line 54, in _reraise_errors     reraise(wrapper, wrapper(exc), sys.exc_info()[2])   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/vine/five.py", line 178, in reraise     raise value.with_traceback(tb)   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors     yield   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps     payload = encoder(data)   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/utils/json.py", line 72, in dumps     **dict(default_kwargs, **kwargs))   File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 238, in dumps     **kw).encode(obj)   File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 199, in encode     chunks = self.iterencode(o, _one_shot=True)   File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 257, in iterencode     return _iterencode(o, 0)   File "/Users/astronaut/PycharmProjects/using_celery_with_flask/venv/lib/python3.6/site-packages/kombu/utils/json.py", line 62, in default     return super(JSONEncoder, self).default(o)   File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 180, in default     o.__class__.__name__) kombu.exceptions.EncodeError: Object of type 'Message' is not JSON serializable

It is highly recommended to not use pickle as it has a vulnerability. And I think that's why it has been replaced in version 4 by JSON. Check the official docs here. Also check out this blogpost discussing the security vulnerability found in pickle.

@Habush would you be interested in crafting a PR that changes this project to use the 3rd option I indicated above? Happy to merge a PR that upgrades to Celery 4 and uses JSON instead of pickle.
That said, the warning that you refer reads "Never unpickle data received from an untrusted or unauthenticated source". The key there is that the data needs to come from an untrusted source. In this application you are running the web server and the Celery workers in a protected environment of a production server. For this to be an issue you would need the attacker to be able to somehow post a malicious message to the message broker (Redis or Rabbit), which is not going to be accessible from the Internet (or at least it shouldn't be). So while I fully agree that it's best to not use pickle, in this particular case the risk is extremely low.

This PR fix this error via 3rd option: https://github.com/miguelgrinberg/flask-celery-example/pull/20

This example was written for Celery 3, which uses pickle as a default serialization format. In Celery 4, the default was changed to JSON, so the error here is that there is no way to serialize the Message object in JSON format.
You can fix the problem in a few ways:

downgrade to Celery v3
configure your Celery v4 to use pickle as serialization format
don't construct a Message instance, just send all the components of the message in a dictionary, then construct the Message instance in the task.

Hi Miguel,
I struggling with this error because of the app_context. I'm starting from the send_async_email and send_email from your wonderful Flask Tutorial. Because of this JSON serialisation thing, I do not find a way to rewrite the code for using celery...
Would you be so kind as to expand your Celery/Flask example with Celery 4/JSON and current_app._get_current_object() (or the right solution)?
I would be eternally grateful and probably other would be too.

@cdruet the code in this repository has been updated to work with Celery 4. Have you seen it?

@miguelgrinberg Just did ;-)
But, as I found out on my own late during the night, it only fixes the Message serialisation thing. I kept struggling with the app_context for a while after that. I ultimately found inspiration in your flasky-with-celery-example on how to structure things to make it work. In doing so, I struggleg with ENV variables when running celery_worker.py (I forgot that I was using dotenv as you suggested in your flasky tutorial, and that it should also be integrated in celery_worker.py).
I'm still convinced the whole thing is worth a post to illustrate where and how things should be declared and organised in a neat app structure.


branch management policy

Following private communication with @slivingston, we may want to select a default fate for branches that have been merged:
I think that deleting a branch associated with a pull request by default is a good practice, unless the branch has been explicitly protected by:

an email announcing so in the tulip mailing list
a comment in the relevant pull request
private communication
branch naming convention

While searching about No.4, I came across the use of slashes in branch names.
I think that the branch name need not be constrained, because it will just become longer.
Of course, if the branch author prefers a particular naming convention, e.g., pr_*, then they can choose to do so.
No.1 is probably the best practice, considering this as logistics extraneous to the PR comments.

I support the proposed policy. Because it concerns merged branches, restoring the branch is easy in cases of mistaken deletion.

Having recorded the discussion here, let us use this approach, until otherwise decided.