fix: Interpreter error formats, better handling

Bug
There is a concern that error resposne format from any connector can change and current error response parsing has no good fallback.
Reviewing
First commit has the main implementation show casing usage on Pipeliner. Consequent commits update the rest of connectors.
Implementation
As of now any error response will go via interpretJSONError method. Every connector has it.
Format switch allows to select the best response format that knows how to convert itself into an error.
schema, err := interpreter.NewFormatSwitch(
 []interpreter.FormatTemplate{
  {
    MustKeys: []string{"description"},
    Template: &DescriptiveResponseError{},
  }, {
    MustKeys: []string{"status", "code"},
    Template: &StatusResponseError{},
  }, {
    MustKeys: nil,
    Template: &MessageResponseError{},
  },
 }...,
).ParseJSON(responseBody)

This can be understood as follows: if response payload has description key then DescriptiveResponseError will be used for JSON parsing and it knows how to produce go error. If desription is missing then it will look for both status and code keys and will then select StatusResponseError format. And lastly it will default to MessageResponseError.
If at any step JSON marshalling fails it will fallback to template that uses the whole response.Body as a message string.
Conclusion, every message is exposed and if lucky it will use good formating.

@Cobalt0s overall looks great, but I think we can further optimize this to reduce number of lines and manual coding errors.
We we define statusCodeMapping as map format and add those in initialization time in NewConnector() call. Same for format interpreter as below
func NewConnector() {
  conn := &SalesloftConnector{}
  ...
  ...
  //some init goes here
  ...
  conn.statusCodeMap = map[http.StatusCode] error
  conn.errorTemplates = []Templates{someTemplates...,...}

}

Also we define statusCodeMapping only once for all connector as below
func statusCodeMapping[T Connector] (conn *T, res *http.Response, body []bytes) error {
    if conn.statusCodeMap == nil {
      return interpreter.DefaultStatusInterpreter(res, body)
    }

    mappedErr, ok := conn.statusCodeMap[res.StatusCode]
    if !ok {
      return interpreter.DefaultStatusInterpreter(res, body)
    }

    return mappedErr

}

The for all of the interpretJSONError function will look the same as below
func (conn *SalesloftConnector) interpretJSONError (res, body) error {
  schema,err := conn.GetErrorFormatter().ParseJSON(body)
  if err != nil {
   return err
  }

  return schema.CombineErr(statusCodeMapping[SalesloftConnector](conn, res, body))

}

In fact we can also improve this by doing below
func (conn *SalesloftConnector) interpretJSONError (res, body) error {
  return InterpretJsonError[SalesloftConnector](conn, res, body)
}

Does this kind of approach make sense to you? This will reduce number of repetitive lines for all connectors and will reduce human code error.
Let me know how you think about this approach.

@jlimatampersand this is great, I like your thoughts! I revisited the code and introduced further code optimization that you have suggested.
New commits
First of all, this commit addresses your general comments: https://github.com/amp-labs/connectors/pull/840/commits/dd771e5f93466d5c1e0c0a695f0ef3a8862a4388
Then I altered some interpreter interfaces: https://github.com/amp-labs/connectors/pull/840/commits/c430f2819261be579294ac097c095df7ab1ab021
I have not used any generics. One problem is that interpreter package shouldn't know/manage data that lives on connector struct. Hence, I took close direction. Under this commit you will also see the usage on Pipeliner.
Next commits follow what Pipeliner did. I have removed some test files errors_test.go as I don't see significant value in them. This is when I introduced any testing and read_test.go and other do better job at covering the error handling. Maybe interpreter package could have more tests.
Connector Init
The connector setup looks as follows:
conn.Client.HTTPClient.ErrorHandler = interpreter.ErrorHandler{
	JSON: interpreter.NewFaultyResponder(errorFormats, statusCodeMapping),
}.Handle

Note: HTTPClient is coming from WithClient(). Maybe we could think how to take this connector parameter and configure with error handling, base url in more readable form. It is refactored code from way ago, and it seems we can do better.

Alternative connector initialization where error is handled in a non uniform way (salesforce):
conn.Client.HTTPClient.ErrorHandler = interpreter.ErrorHandler{
	JSON: &interpreter.DirectFaultyResponder{Callback: conn.interpretJSONError},
	XML:  &interpreter.DirectFaultyResponder{Callback: conn.interpretXMLError},
}.Handle

Benefits
For people that will implement a connector it will be a matter of specifying:

list of possible error formats (optional, but will yield better message formating control)
unique status code mappings (if any)
This means only data driven, so configuration like.
By design, I kept the handler as interface so anybody can plug in their implementation, just in case.

LGTM! thanks for working on this!


Release azure-cli-vm 2.0.2

Reviewers, please verify the following for this PR:

[ ] The PR title has format Release <component-name> <version>.
[ ] setup.py has been modified with the same version as in the PR title.
[ ] HISTORY.rst has been modified with appropriate release notes.
[ ] If releasing azure-cli or azure-cli-core, __version__ defined in __init__.py should also be modified to match.

Started releasing azure-cli-vm 2.0.2...

Release of 'azure-cli-vm' with version '2.0.2' successful. View at https://pypi.python.org/pypi/azure-cli-vm/2.0.2.

GitHub release created. https://github.com/Azure/azure-cli/releases.


Fix warnings reported by clang

Mostly related to override and one to string concatenation in an exception

Thanks again! :+1:

My pleasure :laughing:


ESP not working

Title says it and the aimbot doesn't work so I am guessing the signatures are outdated.

Working after https://github.com/alza54/opensource2/commit/7dbb3f4b7b34131da3f543726c3dd989307fff98
Don't use manual mapping if you don't wish to modify the hooking method

The base way is to Manual map and edit hooking methods. LdrpLoadDll otherwise maybe


#14245: Fix the L1 memory alignment issue caused by DRAM-L1 memory copy. Added the flag in mem_config to indicate memory alignment in L1

Ticket
Link to Github Issue
https://github.com/tenstorrent/tt-metal/issues/14245
Problem description
memory copy from DRAM to L1 is not well aligned to 16B, when the shard length is not 32B aligned but 16B aligned, there are gaps in L1.
What's changed
Added flag in memory_config data structure to indicate the alignment L1; Use the flag to decide the alignment in L1 for i2s and s2i ops. The update can also address issues for some ops that modifies the tensor size in L1 such as maxpooling, dilution.
Checklist

[x] Post commit CI passes
https://github.com/tenstorrent/tt-metal/actions/runs/12110638689
[x] Blackhole Post commit (if applicable)
https://github.com/tenstorrent/tt-metal/actions/runs/12110643661
[ ] Model regression CI testing passes (if applicable)
[ ] Device performance regression CI testing passes (if applicable)
[ ] New/Existing tests provide coverage for changes

Is the issue with not removing the blackhole specific check because the op tests are shared between BH and other arches? Don't quite understand why it's still needed instead of setting keep_l1_aligned, but not a blocker.

BH is always L1 aligned. But WH and other arches are not depending on the src and dst. Ideally we will decide by it's memory  type but I need to confirm 2 things:

That's the intended behavior.
It won't break anything, including unit tests.

So a quick fix is to put a flag for the moment. We can make the decision later.


Update build tools to 25.0.3

Since the travis build has to do with iOS only I can merge this into the Android samples.


Mast: Watch interiorworks.ae

Mast requests the watch of the watch_keyword interiorworks\.ae. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

interiorworks\.ae has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by Daniil in Charcoal HQ


Only do a pull every N weeks

Right now, we do a docker pull every time we run the CLI just to see if we have a new version. Instead, we could set this up to only run if it hasn't attempted a pull in N day.

Relevant code is here, function OutOfDate.
https://github.com/google/shipshape/blob/master/shipshape/util/docker/docker.go


Why print expired error with delta time?

Why print expired error with delta time?
Code

the delta represents how it's been since the token expired

@swchoi727 I understood it, but with this delta time in error message difficult to handle error into code :(


@conda-forge-admin, please add user @messense

Additional comment:
No response

Looks like the bot is not responding, let me try it in a comment.

@conda-forge-admin, please add user @messense


Feat(File Explorer): indicate which files have errors (parse or reconcile)

For any file that has a reconcile or parse error, color the file path red.
Figma mocks here

Done as a part of https://github.com/rilldata/rill/pull/4856


Consider cleaning up preprocessor/normalizer interfaces (breaking changes)

As discussed in https://github.com/deeplearning4j/nd4j/pull/1547, with the introduction of normalization for MultiDataSet, the interface ecosystem has become a bit messy. There's quite some duplication between the interfaces, but unifying them would incur some breaking changes. We could consider this for 0.8.0.
The interface structure right now:
DataSetPreProcessor -> DataNormalization

MultiDataSetPreProcessor -> MultiDataNormalization

With the use of generics, these could be (partially) merged. Also, we could consider making method names more consistent here and there and removing redundant methods.

Next release is 0.8.0 so this might be a good pull request.

How much time do I have? :P

We are doing code freeze in about a week or so. If you want to get this in great if you can't it's not an issue.

Alright, I'll see what I can do. Sounds like it'll be right in time to get 0.8.0 in my production stack I'm building right now, so this would be nice to have for myself too.

@EdeMeijer updates here?

Sort of. I poked around but soon realized it's not as straightforward as I thought it would be. The biggest issue is that DataSet normalizers work with a single INDArray for features and labels while MultiDataSet normalizers work with INDArray[] arrays everywhere. This is not something that can be easily generalized with generics.
However, if there's still time, there's something else I'd like to do. I reworked the whole normalizer serde stuff to support all multi normalizer types and package everything in single files, but I realized it would be better to encode the normalizer's type in the serialized data, so it can be restored without having to know the type beforehand. This way, we could go back from a MultiNormalizerStandardizeSerializer, MultiNormalizerMinMaxScalerSerializer etc to just a NormalizerSerializer facade that works transparently for all types. Also, I'm missing a restore(InputStream) (only implemented restore(File)).
I think I can finish this in a few hours and should be a fairly safe change.

See also #1425 #1650

@AlexDBlack @eraly  is this still an issue? I remember there were normalizer-related changed some time ago...

Yeah, there was a PR that improved functionality for normalizers a long time ago.


ICMP type programmed as a string

New rules command programs the icmp_type as a string but felix rejects that at validation time.  It should be an int.

Fixed under #236


Error on create C# script

Godot version:

v3.0.3
OS/device including version:

Fedora Linux 28 x64 bits 4.16.14-300.fc28.x86_64
Mono JIT compiler version <IP_ADDRESS> (tarball Tue May  8 09:28:02 UTC 2018)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug
	Interpreter:   yes
	LLVM:          supported, not enabled.
	GC:            sgen (concurrent by default)

Issue description:

 modules/mono/editor/godotsharp_builds.cpp:135 - Cannot find binary for 'MSBuild (Mono)'
 The build method threw an exception.
System.IO.FileNotFoundException: Cannot find the MSBuild executable.
 modules/mono/csharp_script.cpp:2360 - Cannot add res://Sprite.cs to the C# project because it could not be created.
 modules/mono/csharp_script.cpp:1957 - Condition ' !valid ' is true. returned: __null

Steps to reproduce:
Install Mono 5.12
Create a Godot Project > Add a C# script.
Minimal reproduction project:

New project with C# script.

Cannot find binary for 'MSBuild (Mono)'

Do you have MSBuild installed? If so, which version?

Should not come with Mono installation?

I did not try install explicity msbuild. Later at home I will try.

mono-complete didn't install msbuild. I had to run explicitly this command:
sudo dnf install msbuild
And works now. Maybe it can be a requirement in the website.


ParamsSpecs: Make serialized form more concise

Write custom serialization / deserialization code so that we don't get the extra layer of indentation.
Current:
app_download: !Value
  value:
    src: https://github.com/azriel91/web_app/releases/download/0.1.1/web_app.tar
    dest: azriel91/web_app/0.1.1/web_app.tar
    marker: null
s3_bucket: !FieldWise
  field_wise_spec:
    name: !Value
      value: azriel-peace-envman-demo-1
    marker: null

Desired:
app_download: !Value
  src: https://github.com/azriel91/web_app/releases/download/0.1.1/web_app.tar
  dest: azriel91/web_app/0.1.1/web_app.tar
s3_bucket: !FieldWise
  name: !Value
    value: azriel-peace-envman-demo-1

Tuple params fieldwise specs are tricky.
Current:
something: !FieldWise
  field_wise_spec: !MappingFn
    field_name: _0
    fn_map: Some(Fn(&bool, &u16) -> Option<Vec<u8>>)
    marker: null

Desired:
something: !FieldWise
  _0: !MappingFn
    fn_map: Some(Fn(&bool, &u16) -> Option<Vec<u8>>)

May be too magic to do this. Tried a few approaches:

#[serde(flatten)] on ParamsSpec::<T>::Value and ParamsSpec::FieldWise:

does not work if T has unnamed fields.
message: Error("can only flatten structs and maps (got a sequence)").

manually serializing ParamsSpec:
Started on this, but it's weird when you get to ParamsSpec::Value, ideally you serialize self as an enum variant for the !Tag, but then get T to serialize its fields, which I don't think can be done:
impl<T> Serialize for ParamsSpec<T>
where
    T: Params + Clone + Debug + Serialize + Send + Sync + 'static,
{
    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
    where
        S: Serializer,
    {
        match self {
            ParamsSpec::Stored => serializer.serialize_unit_variant("ParamsSpec", 0, "Stored"),
            ParamsSpec::Value { value } => {
                match T::type_kind() {
                    TypeKind::Unit => serializer.serialize_unit_variant("ParamsSpec", 1, "Value"),
                    TypeKind::Tuple => serializer.serialize_tuple_variant("ParamsSpec", 1, "Value", value.fields_unnamed()),
                    TypeKind::Struct => serializer.serialize_struct_variant("ParamsSpec", 1, "Value", value.fields_named()),
                    TypeKind::Enum => todo!(),
                }
            }
            ParamsSpec::InMemory => serializer.serialize_unit_variant("ParamsSpec", 2, "InMemory"),
            ParamsSpec::MappingFn(value) => serializer.serialize_newtype_variant("ParamsSpec", 3, "MappingFn", value),
            ParamsSpec::FieldWise { field_wise_spec } => todo!(),
        }
    }
}

/// The kind of type this is.
///
/// Needed to determine how to serialize/deserialize a type at runtime.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum TypeKind {
    /// Unit struct with no fields.
    Unit,
    /// Tuple struct with unnamed fields.
    Tuple,
    /// Tuple struct with named fields.
    Named,
    /// Enum.
    Enum,
}


Allow more primes (for Circom v2.1.8)

Circom v2.1.8 supports: bn128, bls12381, goldilocks, grumpkin, pallas, vesta, secq256r1

A version map could also be made here, comparing the given prime to the configured version (since main component's Circom file will have that version on it)


Is the artifactory repository down?

Yesterday, when trying to build using your library it threw this error:
> Could not resolve all files for configuration ':app:debugRuntimeClasspath'.
   > Could not resolve com.nike.epc:epc-core:1.0.0.
     Required by:
         project :app > project :zebra_rfid
      > Could not resolve com.nike.epc:epc-core:1.0.0.
         > Could not get resource 'https://artifactory.nike.com/artifactory/maven/com/nike/epc/epc-core/1.0.0/epc-core-1.0.0.pom'.
            > Could not GET 'https://artifactory.nike.com/artifactory/maven/com/nike/epc/epc-core/1.0.0/epc-core-1.0.0.pom'.
               > artifactory.nike.com: nodename nor servname provided, or not known

I have been using your library for several weeks now, but just yesterday it failed.

Unfortunately this is the result of the bintray deprecation. We are no longer publishing to bintray and for the time being we are using an internal-only artifactory server to host for projects within Nike. There are plans to address this and get publishing working globally again, but I'm unsure when it will happen.

Any chance you could instead publish it to Maven Central?

@PabloReszczynski I'm not sure what the long term plans are for publishing. In the meantime if you checkout the repo and run sbt publishLocal you should be able to depend on 1.0.2-SNAPSHOT for local development at least. That might not be suitable for production deployments though.


serializer option to disable data and attributes scoping

we'd like to switch from AMS to using fast_jsonapi but we can't customize the json output the way we need to.
our APIs are currently consumed, we can't just ask all api consumer to switch output format because we decided to use this gem.
fact is AMS let us output any json output we want while this gem is opinionated about the structured
we want to return simple structure such as
{"a":"11002458-818b-4ab8-a190-668630b4aed4","b":"815279f3-59d2-434a-9462-da87599840c5","c":0.0,"d":null}
not
{"data":{"attributes":{"a":"11002458-818b-4ab8-a190-668630b4aed4","b":"815279f3-59d2-434a-9462-da87599840c5","c":0.0,"d":null}}}
thank you

fast_jsonapi follows JSON API format, and unfortunately I do not think you can change the format. However the good new is that AMS is not that slow compared with fast_jsonapi as long as you use default adaptor. See https://github.com/Netflix/fast_jsonapi/issues/118 👍

Ah

we fixed AMS to FJA format as below by overriding serializable_hash
  def serializable_hash
    data = super

    if data[:data].is_a? Hash
      data[:data][:attributes]

    elsif data[:data].is_a? Array
      data[:data].map{ |x| x[:attributes] }

    elsif data[:data] == nil
      nil

    else
      data
    end
  end

we fixed AMS to FJA format as below by overriding serializable_hash
  def serializable_hash
    data = super

    if data[:data].is_a? Hash
      data[:data][:attributes]

    elsif data[:data].is_a? Array
      data[:data].map{ |x| x[:attributes] }

    elsif data[:data] == nil
      nil

    else
      data
    end
  end

@amolpujari what file I put your code to ? Can you give me a example ?

This is really sweet thank you.
When I get a chance. I might turn this into an Option

# app/serializers/application_serializer.rb

class ApplicationSerializer
  include FastJsonapi::ObjectSerializer
  #cache_options enabled: true, cache_length: 1.hour

  def to_h
    data = serializable_hash

    if data[:data].is_a? Hash
      data[:data][:attributes]

    elsif data[:data].is_a? Array
      data[:data].map{ |x| x[:attributes] }

    elsif data[:data] == nil
      nil

    else
      data
    end
  end

  class << self
    def has_one resource, options={}
      serializer = options[:serializer] || "#{resource.to_s.classify}Serializer".constantize

      attribute resource do |object|
        serializer.new(object.try(resource)).to_h
      end
    end

    def has_many resources, options={}
      serializer = options[:serializer] || "#{resources.to_s.classify}Serializer".constantize

      attribute resources do |object|
        serializer.new(object.try(resources)).to_h
      end
    end
  end
end


Missing 1.3.6 RPM in yum repo

InfluxDB 1.3.6 was released 19 days ago but the official repo still does not have the rpm.

Also the changelog shows last release as 1.3.3.

Version 1.3.7 in the repo https://repos.influxdata.com/rhel/7/x86_64/stable/influxdb-1.3.7.x86_64.rpm
Thanks!


Info/help layer update

Includes new contrboard features and required visual changes like better slide indicator on bottom

Run gulp lint for the changed files

:smoking:


problem with original vs used title for foreign movies

defaults to original title (no akas)
example: https://www.imdb.com/title/tt8963328/
Hi Score Girl
Hai Sukoa Garu (original title)

just wanted to add, you do awesome work. thanks for the effort.

if you want the original title use getTitle($bForceLocal = true)

brilliant! thanks :D


Fantasy Land compatibility

Do you have any plans to make funfix compatible with Fantasy Land for interoperability with other functional libraries like ramda? I believe a lot of the functions for compatibility already exist in funfix, but functions would need to be aliased to conform to the Fantasy Land prefixed method names. Thanks for the great library!

@mrdziuban the answer is yes, there are plans.
But I have to get familiar with Fantasy Land and I'll probably need help for it.
Will leave this ticket open to track the progress made.

Thanks @alexandru, I'll keep an eye on this ticket. I'm happy to help where possible though I would need to get more familiar with Fantasy Land as well.

Fantasy-Land compatibility has just landed on master, a pre-release version will get published soon.

Fantasy Land spec is violated by things like Option.of checking for nulls.


Some undefined variables in the new code

Hi,
@tranluan the new commits posted last week. I tried running the code and I'm getting some variables that aren't defined. Do you know what edits we need to make to fix it?
Input: python main_non_linear_3DMM.py
Output
{'batch_size': 64,
'beta1': 0.5,
'c_dim': 3,
'checkpoint_dir': 'checkpoint',
'dataset': 'celebA',
'df_dim': 32,
'dfc_dim': 512,
'epoch': 1000,
'gf_dim': 32,
'gfc_dim': 512,
'gpu': '0',
'h': False,
'help': False,
'helpfull': False,
'helpshort': False,
'image_size': 108,
'is_batchwise_white_shading': False,
'is_const_albedo': False,
'is_const_local_albedo': False,
'is_crop': False,
'is_gt_m': False,
'is_partbase_albedo': False,
'is_pretrain': False,
'is_reduce': False,
'is_smoothness': False,
'is_train': True,
'is_using_frecon': False,
'is_using_graddiff': False,
'is_using_landmark': False,
'is_using_linear': False,
'is_using_recon': False,
'is_using_symetry': False,
'is_with_y': True,
'learning_rate': 0.0002,
'output_size': 224,
'sample_size': 64,
'samples_dir': 'samples',
'train_size': 5000000,
'visualize': False,
'z_dim': 50}
2019-07-22 11:46:36.925753: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-22 11:46:37.039450: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-22 11:46:37.040339: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5a2dbb0 executing computations on platform CUDA. Devices:
2019-07-22 11:46:37.040355: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): GeForce RTX 2060, Compute Capability 7.5
2019-07-22 11:46:37.062342: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-07-22 11:46:37.062777: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5a97540 executing computations on platform Host. Devices:
2019-07-22 11:46:37.062796: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): ,
2019-07-22 11:46:37.062964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.2
pciBusID: 0000:01:00.0
totalMemory: 5.77GiB freeMemory: 4.55GiB
2019-07-22 11:46:37.062980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-07-22 11:46:37.065177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-22 11:46:37.065190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0
2019-07-22 11:46:37.065199: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N
2019-07-22 11:46:37.065300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4723 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5)
Loading 3DMM tri ...
DONE
Loading 3DMM vertex tri ...
DONE
Loading ./3DMM_definition/3DMM_shape_basis.dat ...
DONE
Loading ./3DMM_definition/3DMM_exp_basis.dat ...
DONE
Making make_parallel for 1 gpu(s)
WARNING:tensorflow:From /home/rayquaza/.local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py:3632: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
[64, 102]
[64, 102]
[64, 256]
WARNING:tensorflow:From /home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/ops.py:128: conv2d_transpose (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d_transpose instead.
[64, 256]
Loading 3DMM tri ...
DONE
Loading 3DMM vertex tri ...
DONE
WARNING:tensorflow:From /home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/rendering_ops.py:544: calling l2_normalize (from tensorflow.python.ops.nn_impl) with dim is deprecated and will be removed in a future version.
Instructions for updating:
dim is deprecated, use axis instead
sh.get_shape()
(64, 43008, 1, 9)
L1.get_shape()
(64, 43008, 9, 1)
Loading 3DMM tri ...
DONE
Loading 3DMM vertex tri ...
DONE
Loading 3DMM keypoints ...
Loading 3DMM keypoints ...
WARNING:tensorflow:From /home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/ops.py:222: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
Loading AFW ...
DONE. Finish loading AFW with 5207 images
Loading AFW_Flip ...
DONE. Finish loading AFW_Flip with 5207 images
Loading HELEN ...
DONE. Finish loading HELEN with 37676 images
Loading HELEN_Flip ...
DONE. Finish loading HELEN_Flip with 37676 images
Loading IBUG ...
DONE. Finish loading IBUG with 1786 images
Loading IBUG_Flip ...
DONE. Finish loading IBUG_Flip with 1786 images
Loading LFPW ...
DONE. Finish loading LFPW with 16556 images
Loading LFPW_Flip ...
DONE. Finish loading LFPW_Flip with 16556 images
Traceback (most recent call last):
File "main_non_linear_3DMM.py", line 87, in
tf.app.run()
File "/home/rayquaza/.local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "main_non_linear_3DMM.py", line 70, in main
dcgan.train(FLAGS)
File "/home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/model_non_linear_3DMM.py", line 405, in train
print("Valid images %d/%d" % ( len(valid_300W_idx), self.images_300W.shape[0] ))
NameError: global name 'valid_300W_idx' is not defined

Sorry. It's just a typo, please replace 'valid_300W_idx' by 'valid_idx'.

Thanks for the quick response! This is what I get now:
print("Valid images %d/%d" % ( len(valid_idx), self.images_300W.shape[0] ))
AttributeError: 'DCGAN' object has no attribute 'images_300W
Also if you would like, once I get this working, I can submit a pull request with all the bugs fixed and documentation added for everyone else.

self.images_300W is self.image_filenames
Please submit a pull request. I appreciate it.

Ya, I figured that one out, now I'm stuck on
ffeed_dict[self.texture_labels_fn_300W[i]] = _300W_LP_DIR + 'texture/'+ image2texture_fn(batch_300W_images_fn[i])
NameError: global name 'image2texture_fn' is not defined

The function image2texture_fn try to convert image filename into texture filename. Since multiple images reuse the same ground truth texture.
Sorry. That's way too many errors that shouldn't be there.
Please add this to utils.py
def image2texture_fn(image_fn):
last = image_fn[-7:].find('_')
if (last < 0):
return image_fn
else:
return image_fn[:-7 + last] + '_0.png'

Ok great, now it's training. So I guess the next question is how can I make sure that it reproduces the results in the paper? Is there a testing script? or a online pretrained model we can compare against?

The weights here are set the same as my most recent experiment.
I usually run pretrain stage for 1 days (~15 epochs) on 4 TitanX GPU with all losses. Then the weight for S and m are set to 0 or reduced significantly.
I don't have access to GPUs at the moment to validate all the hyper parameters on this "clean up" implementation. That's why I was hesitating posting the code. Gonna post the command to run the code soon.

Can you share with me the command to run the pretrained model?

python main_non_linear_3DMM_lighting_all2d_hybrid.py --batch_size 128 --sample_size 128 --is_train True --learning_rate 0.001 --ouput_size  224 \
   --gf_dim 32 --df_dim 32 --dfc_dim 320 --gfc_dim 320 --z_dim 20 --c_dim 3 \
   --is_using_landmark True  --shape_loss l2 --tex_loss l1 \
   --is_using_recon False --is_using_frecon False --is_partbase_albedo False --is_using_symetry True  \
   --is_albedo_supervision False --is_batchwise_white_shading True --is_const_albedo True --is_const_local_albedo True --is_smoothness True
  --gpu 0,1,2,3 \

This is the code to train, what about to just test out the model on some images and visualize those results?

You can just simply change the network setting is_train = False, then feed the image to the network. You can use any library to visualize the 3D shape (topology is given in tri file).
Maybe I'll add later if I have time.

@tranluan Can you share with me the command to test the model,Thank you

@tranluan What is the path to the weights? The checkpoint path is not found as it looks to me.

line 73, in main
dcgan.test(FLAGS, True)
AttributeError: 'DCGAN' object has no attribute 'test'
^ What do I do here?

@KalraA  what is your tensorflow  and cuda version? I have tried to run this code, but encounter an error of " [[ZbufferTriV2Sz224]]
Errors may have originated from an input operation.
Input Source operations connected to node ZbufferTriV2Sz224:"

python main_non_linear_3DMM.py --batch_size 128 --sample_size 128 --is_train True --learning_rate 0.001 --ouput_size  224 \
   --gf_dim 32 --df_dim 32 --dfc_dim 320 --gfc_dim 320 --z_dim 20 --c_dim 3 \
   --is_using_landmark True  --shape_loss l2 --tex_loss l1 \
   --is_using_recon False --is_using_frecon False --is_partbase_albedo False --is_using_symetry True  \
   --is_albedo_supervision False --is_batchwise_white_shading True --is_const_albedo True --is_const_local_albedo True --is_smoothness True
  --gpu 0,1,2,3 \

Turn off the "--is_using_recon", "--is_using_frecon" could speed up the training, (as the rendering layer is slow T_T). #1

loss is NAN using above command

Hi,
@tranluan the new commits posted last week. I tried running the code and I'm getting some variables that aren't defined. Do you know what edits we need to make to fix it?
Input: python main_non_linear_3DMM.py
Output
{'batch_size': 64,
'beta1': 0.5,
'c_dim': 3,
'checkpoint_dir': 'checkpoint',
'dataset': 'celebA',
'df_dim': 32,
'dfc_dim': 512,
'epoch': 1000,
'gf_dim': 32,
'gfc_dim': 512,
'gpu': '0',
'h': False,
'help': False,
'helpfull': False,
'helpshort': False,
'image_size': 108,
'is_batchwise_white_shading': False,
'is_const_albedo': False,
'is_const_local_albedo': False,
'is_crop': False,
'is_gt_m': False,
'is_partbase_albedo': False,
'is_pretrain': False,
'is_reduce': False,
'is_smoothness': False,
'is_train': True,
'is_using_frecon': False,
'is_using_graddiff': False,
'is_using_landmark': False,
'is_using_linear': False,
'is_using_recon': False,
'is_using_symetry': False,
'is_with_y': True,
'learning_rate': 0.0002,
'output_size': 224,
'sample_size': 64,
'samples_dir': 'samples',
'train_size': 5000000,
'visualize': False,
'z_dim': 50}
2019-07-22 11:46:36.925753: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-22 11:46:37.039450: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-22 11:46:37.040339: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5a2dbb0 executing computations on platform CUDA. Devices:
2019-07-22 11:46:37.040355: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce RTX 2060, Compute Capability 7.5
2019-07-22 11:46:37.062342: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-07-22 11:46:37.062777: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5a97540 executing computations on platform Host. Devices:
2019-07-22 11:46:37.062796: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2019-07-22 11:46:37.062964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.2
pciBusID: 0000:01:00.0
totalMemory: 5.77GiB freeMemory: 4.55GiB
2019-07-22 11:46:37.062980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-07-22 11:46:37.065177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-22 11:46:37.065190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-07-22 11:46:37.065199: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-07-22 11:46:37.065300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4723 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5)
Loading 3DMM tri ...
DONE
Loading 3DMM vertex tri ...
DONE
Loading ./3DMM_definition/3DMM_shape_basis.dat ...
DONE
Loading ./3DMM_definition/3DMM_exp_basis.dat ...
DONE
Making make_parallel for 1 gpu(s)
WARNING:tensorflow:From /home/rayquaza/.local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py:3632: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
[64, 102]
[64, 102]
[64, 256]
WARNING:tensorflow:From /home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/ops.py:128: conv2d_transpose (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d_transpose instead.
[64, 256]
Loading 3DMM tri ...
DONE
Loading 3DMM vertex tri ...
DONE
WARNING:tensorflow:From /home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/rendering_ops.py:544: calling l2_normalize (from tensorflow.python.ops.nn_impl) with dim is deprecated and will be removed in a future version.
Instructions for updating:
dim is deprecated, use axis instead
sh.get_shape()
(64, 43008, 1, 9)
L1.get_shape()
(64, 43008, 9, 1)
Loading 3DMM tri ...
DONE
Loading 3DMM vertex tri ...
DONE
Loading 3DMM keypoints ...
Loading 3DMM keypoints ...
WARNING:tensorflow:From /home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/ops.py:222: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
Loading AFW ...
DONE. Finish loading AFW with 5207 images
Loading AFW_Flip ...
DONE. Finish loading AFW_Flip with 5207 images
Loading HELEN ...
DONE. Finish loading HELEN with 37676 images
Loading HELEN_Flip ...
DONE. Finish loading HELEN_Flip with 37676 images
Loading IBUG ...
DONE. Finish loading IBUG with 1786 images
Loading IBUG_Flip ...
DONE. Finish loading IBUG_Flip with 1786 images
Loading LFPW ...
DONE. Finish loading LFPW with 16556 images
Loading LFPW_Flip ...
DONE. Finish loading LFPW_Flip with 16556 images
Traceback (most recent call last):
File "main_non_linear_3DMM.py", line 87, in
tf.app.run()
File "/home/rayquaza/.local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "main_non_linear_3DMM.py", line 70, in main
dcgan.train(FLAGS)
File "/home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/model_non_linear_3DMM.py", line 405, in train
print("Valid images %d/%d" % ( len(valid_300W_idx), self.images_300W.shape[0] ))
NameError: global name 'valid_300W_idx' is not defined
hi, I try to use the author provide   command to run the main_nonlinear_3DMM.py ,but meet the follow error,  can you help me to complain this problem?  thank you very much.
NotFoundError: ./TF_newop/cuda_op_kernel_v2_sz224.so: cannot open shared object file: No such file or directory #41
the command as follow:
python main_non_linear_3DMM.py --batch_size 128 --sample_size 128 --is_train True --learning_rate 0.001 --ouput_size  224
--gf_dim 32 --df_dim 32 --dfc_dim 320 --gfc_dim 320 --z_dim 20 --c_dim 3
--is_using_landmark True  --shape_loss l2 --tex_loss l1
--is_using_recon False --is_using_frecon False --is_partbase_albedo False --is_using_symetry True
--is_albedo_supervision False --is_batchwise_white_shading True --is_const_albedo True --is_const_local_albedo True --is_smoothness True
--gpu 0

Ok great, now it's training. So I guess the next question is how can I make sure that it reproduces the results in the paper? Is there a testing script? or a online pretrained model we can compare against?

Hi @KalraA
I just saw your comments here. It seems that you could train their model. As I have some issues with the rendering module, would you please give me the details of your environment you used for training this code? I'm using ubuntu 18, python 2.7 and tensorflow 1.8 and I'm facing errors when I run the main code. The error I get when I run the rendering_example.py is  as follows:
The error I get when I run rendering_example.py:
InternalError: Blas xGEMMBatched launch failed : a.shape=[16,4,3], b.shape=[16,53215,4], m=3, n=53215, k=4, batch_size=16
[[Node: MatMul = BatchMatMul[T=DT_FLOAT, adj_x=true, adj_y=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](concat, concat_1)]]
[[Node: Reshape_43/_147 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1301_Reshape_43", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Any advice is highly appreciated.

Hi,
@tranluan the new commits posted last week. I tried running the code and I'm getting some variables that aren't defined. Do you know what edits we need to make to fix it?
Input: python main_non_linear_3DMM.py
Output
{'batch_size': 64,
'beta1': 0.5,
'c_dim': 3,
'checkpoint_dir': 'checkpoint',
'dataset': 'celebA',
'df_dim': 32,
'dfc_dim': 512,
'epoch': 1000,
'gf_dim': 32,
'gfc_dim': 512,
'gpu': '0',
'h': False,
'help': False,
'helpfull': False,
'helpshort': False,
'image_size': 108,
'is_batchwise_white_shading': False,
'is_const_albedo': False,
'is_const_local_albedo': False,
'is_crop': False,
'is_gt_m': False,
'is_partbase_albedo': False,
'is_pretrain': False,
'is_reduce': False,
'is_smoothness': False,
'is_train': True,
'is_using_frecon': False,
'is_using_graddiff': False,
'is_using_landmark': False,
'is_using_linear': False,
'is_using_recon': False,
'is_using_symetry': False,
'is_with_y': True,
'learning_rate': 0.0002,
'output_size': 224,
'sample_size': 64,
'samples_dir': 'samples',
'train_size': 5000000,
'visualize': False,
'z_dim': 50}
2019-07-22 11:46:36.925753: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-22 11:46:37.039450: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-22 11:46:37.040339: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5a2dbb0 executing computations on platform CUDA. Devices:
2019-07-22 11:46:37.040355: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce RTX 2060, Compute Capability 7.5
2019-07-22 11:46:37.062342: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-07-22 11:46:37.062777: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5a97540 executing computations on platform Host. Devices:
2019-07-22 11:46:37.062796: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2019-07-22 11:46:37.062964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.2
pciBusID: 0000:01:00.0
totalMemory: 5.77GiB freeMemory: 4.55GiB
2019-07-22 11:46:37.062980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-07-22 11:46:37.065177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-22 11:46:37.065190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-07-22 11:46:37.065199: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-07-22 11:46:37.065300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4723 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5)
Loading 3DMM tri ...
DONE
Loading 3DMM vertex tri ...
DONE
Loading ./3DMM_definition/3DMM_shape_basis.dat ...
DONE
Loading ./3DMM_definition/3DMM_exp_basis.dat ...
DONE
Making make_parallel for 1 gpu(s)
WARNING:tensorflow:From /home/rayquaza/.local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py:3632: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
[64, 102]
[64, 102]
[64, 256]
WARNING:tensorflow:From /home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/ops.py:128: conv2d_transpose (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d_transpose instead.
[64, 256]
Loading 3DMM tri ...
DONE
Loading 3DMM vertex tri ...
DONE
WARNING:tensorflow:From /home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/rendering_ops.py:544: calling l2_normalize (from tensorflow.python.ops.nn_impl) with dim is deprecated and will be removed in a future version.
Instructions for updating:
dim is deprecated, use axis instead
sh.get_shape()
(64, 43008, 1, 9)
L1.get_shape()
(64, 43008, 9, 1)
Loading 3DMM tri ...
DONE
Loading 3DMM vertex tri ...
DONE
Loading 3DMM keypoints ...
Loading 3DMM keypoints ...
WARNING:tensorflow:From /home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/ops.py:222: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
Loading AFW ...
DONE. Finish loading AFW with 5207 images
Loading AFW_Flip ...
DONE. Finish loading AFW_Flip with 5207 images
Loading HELEN ...
DONE. Finish loading HELEN with 37676 images
Loading HELEN_Flip ...
DONE. Finish loading HELEN_Flip with 37676 images
Loading IBUG ...
DONE. Finish loading IBUG with 1786 images
Loading IBUG_Flip ...
DONE. Finish loading IBUG_Flip with 1786 images
Loading LFPW ...
DONE. Finish loading LFPW with 16556 images
Loading LFPW_Flip ...
DONE. Finish loading LFPW_Flip with 16556 images
Traceback (most recent call last):
File "main_non_linear_3DMM.py", line 87, in
tf.app.run()
File "/home/rayquaza/.local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "main_non_linear_3DMM.py", line 70, in main
dcgan.train(FLAGS)
File "/home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/model_non_linear_3DMM.py", line 405, in train
print("Valid images %d/%d" % ( len(valid_300W_idx), self.images_300W.shape[0] ))
NameError: global name 'valid_300W_idx' is not defined

Hi,I occur to the error:
NameError: global name '_300W_LP_DIR' is not defined

Hi,
@tranluan the new commits posted last week. I tried running the code and I'm getting some variables that aren't defined. Do you know what edits we need to make to fix it?
Input: python main_non_linear_3DMM.py
Output
{'batch_size': 64,
'beta1': 0.5,
'c_dim': 3,
'checkpoint_dir': 'checkpoint',
'dataset': 'celebA',
'df_dim': 32,
'dfc_dim': 512,
'epoch': 1000,
'gf_dim': 32,
'gfc_dim': 512,
'gpu': '0',
'h': False,
'help': False,
'helpfull': False,
'helpshort': False,
'image_size': 108,
'is_batchwise_white_shading': False,
'is_const_albedo': False,
'is_const_local_albedo': False,
'is_crop': False,
'is_gt_m': False,
'is_partbase_albedo': False,
'is_pretrain': False,
'is_reduce': False,
'is_smoothness': False,
'is_train': True,
'is_using_frecon': False,
'is_using_graddiff': False,
'is_using_landmark': False,
'is_using_linear': False,
'is_using_recon': False,
'is_using_symetry': False,
'is_with_y': True,
'learning_rate': 0.0002,
'output_size': 224,
'sample_size': 64,
'samples_dir': 'samples',
'train_size': 5000000,
'visualize': False,
'z_dim': 50}
2019-07-22 11:46:36.925753: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-22 11:46:37.039450: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-07-22 11:46:37.040339: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5a2dbb0 executing computations on platform CUDA. Devices:
2019-07-22 11:46:37.040355: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce RTX 2060, Compute Capability 7.5
2019-07-22 11:46:37.062342: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-07-22 11:46:37.062777: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5a97540 executing computations on platform Host. Devices:
2019-07-22 11:46:37.062796: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2019-07-22 11:46:37.062964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce RTX 2060 major: 7 minor: 5 memoryClockRate(GHz): 1.2
pciBusID: 0000:01:00.0
totalMemory: 5.77GiB freeMemory: 4.55GiB
2019-07-22 11:46:37.062980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-07-22 11:46:37.065177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-22 11:46:37.065190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-07-22 11:46:37.065199: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-07-22 11:46:37.065300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4723 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:01:00.0, compute capability: 7.5)
Loading 3DMM tri ...
DONE
Loading 3DMM vertex tri ...
DONE
Loading ./3DMM_definition/3DMM_shape_basis.dat ...
DONE
Loading ./3DMM_definition/3DMM_exp_basis.dat ...
DONE
Making make_parallel for 1 gpu(s)
WARNING:tensorflow:From /home/rayquaza/.local/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py:3632: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
[64, 102]
[64, 102]
[64, 256]
WARNING:tensorflow:From /home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/ops.py:128: conv2d_transpose (from tensorflow.python.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d_transpose instead.
[64, 256]
Loading 3DMM tri ...
DONE
Loading 3DMM vertex tri ...
DONE
WARNING:tensorflow:From /home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/rendering_ops.py:544: calling l2_normalize (from tensorflow.python.ops.nn_impl) with dim is deprecated and will be removed in a future version.
Instructions for updating:
dim is deprecated, use axis instead
sh.get_shape()
(64, 43008, 1, 9)
L1.get_shape()
(64, 43008, 9, 1)
Loading 3DMM tri ...
DONE
Loading 3DMM vertex tri ...
DONE
Loading 3DMM keypoints ...
Loading 3DMM keypoints ...
WARNING:tensorflow:From /home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/ops.py:222: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
Loading AFW ...
DONE. Finish loading AFW with 5207 images
Loading AFW_Flip ...
DONE. Finish loading AFW_Flip with 5207 images
Loading HELEN ...
DONE. Finish loading HELEN with 37676 images
Loading HELEN_Flip ...
DONE. Finish loading HELEN_Flip with 37676 images
Loading IBUG ...
DONE. Finish loading IBUG with 1786 images
Loading IBUG_Flip ...
DONE. Finish loading IBUG_Flip with 1786 images
Loading LFPW ...
DONE. Finish loading LFPW with 16556 images
Loading LFPW_Flip ...
DONE. Finish loading LFPW_Flip with 16556 images
Traceback (most recent call last):
File "main_non_linear_3DMM.py", line 87, in
tf.app.run()
File "/home/rayquaza/.local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "main_non_linear_3DMM.py", line 70, in main
dcgan.train(FLAGS)
File "/home/rayquaza/Documents/faces/Nonlinear_Face_3DMM/model_non_linear_3DMM.py", line 405, in train
print("Valid images %d/%d" % ( len(valid_300W_idx), self.images_300W.shape[0] ))
NameError: global name 'valid_300W_idx' is not defined

Hi,I occur to the error:
NameError: global name '_300W_LP_DIR' is not defined

Ok great, now it's training. So I guess the next question is how can I make sure that it reproduces the results in the paper? Is there a testing script? or a online pretrained model we can compare against?

Hi @KalraA
I just saw your comments here. It seems that you could train their model. As I have some issues with the rendering module, would you please give me the details of your environment you used for training this code? I'm using ubuntu 18, python 2.7 and tensorflow 1.8 and I'm facing errors when I run the main code. The error I get when I run the rendering_example.py is as follows:
The error I get when I run rendering_example.py:
InternalError: Blas xGEMMBatched launch failed : a.shape=[16,4,3], b.shape=[16,53215,4], m=3, n=53215, k=4, batch_size=16
[[Node: MatMul = BatchMatMul[T=DT_FLOAT, adj_x=true, adj_y=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](concat, concat_1)]]
[[Node: Reshape_43/_147 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1301_Reshape_43", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Any advice is highly appreciated.
I run rendering_example.py also occur to the error:
InternalError: Blas xGEMMBatched launch failed : a.shape=[16,4,3], b.shape=[16,53215,4], m=3, n=53215, k=4, batch_size=16
[[Node: MatMul = BatchMatMul[T=DT_FLOAT, adj_x=true, adj_y=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](concat, concat_1)]]
[[Node: Reshape_43/_147 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1301_Reshape_43", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
would you like the share the solution with me?Thanks very much？I am looking forward to your reply!

Ok great, now it's training. So I guess the next question is how can I make sure that it reproduces the results in the paper? Is there a testing script? or a online pretrained model we can compare against?

Hi @KalraA
I just saw your comments here. It seems that you could train their model. As I have some issues with the rendering module, would you please give me the details of your environment you used for training this code? I'm using ubuntu 18, python 2.7 and tensorflow 1.8 and I'm facing errors when I run the main code. The error I get when I run the rendering_example.py is as follows:
The error I get when I run rendering_example.py:
InternalError: Blas xGEMMBatched launch failed : a.shape=[16,4,3], b.shape=[16,53215,4], m=3, n=53215, k=4, batch_size=16
[[Node: MatMul = BatchMatMul[T=DT_FLOAT, adj_x=true, adj_y=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](concat, concat_1)]]
[[Node: Reshape_43/_147 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1301_Reshape_43", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Any advice is highly appreciated.
I run rendering_example.py also occur to the error:
InternalError: Blas xGEMMBatched launch failed : a.shape=[16,4,3], b.shape=[16,53215,4], m=3, n=53215, k=4, batch_size=16
[[Node: MatMul = BatchMatMul[T=DT_FLOAT, adj_x=true, adj_y=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](concat, concat_1)]]
[[Node: Reshape_43/_147 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1301_Reshape_43", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
would you like the share the solution with me?Thanks very much？I am looking forward to your reply!

Ok great, now it's training. So I guess the next question is how can I make sure that it reproduces the results in the paper? Is there a testing script? or a online pretrained model we can compare against?

Hi @KalraA I just saw your comments here. It seems that you could train their model. As I have some issues with the rendering module, would you please give me the details of your environment you used for training this code? I'm using ubuntu 18, python 2.7 and tensorflow 1.8 and I'm facing errors when I run the main code. The error I get when I run the rendering_example.py is as follows: The error I get when I run rendering_example.py: InternalError: Blas xGEMMBatched launch failed : a.shape=[16,4,3], b.shape=[16,53215,4], m=3, n=53215, k=4, batch_size=16 [[Node: MatMul = BatchMatMul[T=DT_FLOAT, adj_x=true, adj_y=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](concat, concat_1)]] [[Node: Reshape_43/_147 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1301_Reshape_43", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Any advice is highly appreciated.

Hi, I've met the same problem. Can you share how you solve it? Thanks.


fix eslint scanned out issues

Hi @andyliuliming, I'm your friendly neighborhood Azure Pull Request Bot (You can call me AZPRBOT). Thanks for your contribution!

It looks like you're working at Microsoft (andliu). If you're full-time, we DON'T require a contribution license agreement.

If you are a vendor, DO please sign the electronic contribution license agreement. It will take 2 minutes and there's no faxing! https://cla.azure.com.

TTYL, AZPRBOT;

@andyliuliming  our apologies for responding to you with a long delay.
I wonder if you would be able to update your PR with the recent code?
If you are willing to do so we will be happy to approve your eslint fix contribution.

Closing because this is stale, but I'll note that we now have tslint support enabled for the v2 version of the SDK and it fails the build if you violate it, so I think we've accomplished the spirit of this PR now. Thanks for the contribution.


Can I use sfttrainer with pytorch dataset

How can I use pytorch's dataset to fine-tune llama3.1.
When I try to use pytorch's dataset, I keep getting the following errors related to collator:
File ~/anaconda3/envs/llama/lib/python3.10/site-packages/transformers/data/data_collator.py:589, in
...
#labels = [feature[label_name] for feature in features] if label_name in features[0].keys() else None
# reconvert list[None] to None if necessary
 # this might occur when we pass {..., "labels": None}

AttributeError: 'str' object has no attribute 'keys'

The reason is that I want to add noise to the word (data-augmentation) and the dataset is dynamic as below.
def __getitem__(self, idx):
        # only add noise to input text
        # tmp = self.data[idx]
        true_qry = [self.data['true_qry'][idx]](url)
        if random.random() < self.noise_prob:
            sample_edit_distance = random.randint(1, self.max_edit_distance)
            input_qry = self.add_noise(true_qry, sample_edit_distance)
        else:
            input_qry = true_qry

And then I follow the fine-tune scipt and use chatml template
<|im_start|>user
iobwin<|im_end|>
<|im_start|>assistant
ibowin<|im_end|>

The trainer is as below:
def my_formatting_func(example):
    return example

trainer=SFTTrainer(
        model=model,
        tokenizer=tokenizer,
        train_dataset=train_dataset,
        # dataset_text_field="text",
        max_seq_length=max_seq_length,
        dataset_num_proc=2,
        packing=False,
        data_collator = DataCollatorForSeq2Seq(tokenizer = tokenizer),
        args=train_args,
        formatting_func=my_formatting_func
    )

Thanks for raising the question @xugy16. In SFTTrainer, the datasets are expected to be datasets.Dataset objects, so you can't use torch dataset.
However, you can add noise in you datasets.Dataset easily, see the following exemple:
from datasets import load_dataset

dataset = load_dataset("timdettmers/openassistant-guanaco")

print(dataset["train"][0]) # {'text': '### Human: Can you write a short introduction [...] ### Human: Now explain it to a dog'}

def apply_noise(example):
    # naive noise addition where we add a [NOISE] to the end of the text
    example["text"] = example["text"] + " [NOISE]"
    return example

noisy_dataset = dataset.map(apply_noise)

print(noisy_dataset["train"][0]) # {'text': '### Human: Can you write a short introduction [...] ### Human: Now explain it to a dog [NOISE]'}

@qgallouedec but this is not dynamic. What if we want to add dynamic noise (changing data for each epoch)


[BEAM-8842] Reactivating BQIO Py test while preventing timeouts.

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Run Python 3.5 PostCommit

r: @udim

Thanks!


bugfix: 同名多配置文件渲染时过滤错误(close #624)

Codecov Report

Merging #628 (f0dcbb7) into V2.2.9-rc (5fc101c) will increase coverage by 0.01%.
The diff coverage is 88.23%.

@@              Coverage Diff              @@
##           V2.2.9-rc     #628      +/-   ##
=============================================
+ Coverage      71.08%   71.10%   +0.01%
=============================================
  Files            292      292
  Lines          21227    21255      +28
=============================================
+ Hits           15090    15113      +23
- Misses          6137     6142       +5

Impacted Files
Coverage Δ

apps/backend/plugin/views.py
36.80% <ø> (ø)

apps/node_man/handlers/plugin_v2.py
85.35% <ø> (ø)

apps/node_man/handlers/policy.py
83.37% <ø> (ø)

apps/backend/plugin/tools.py
90.41% <86.36%> (-0.41%)
:arrow_down:

apps/backend/subscription/steps/adapter.py
89.52% <91.66%> (-0.19%)
:arrow_down:

apps/node_man/handlers/job.py
85.79% <0.00%> (-0.27%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5fc101c...f0dcbb7. Read the comment docs.


Connecting to a host only providing /api

The settings page checks if the response to https://example.com is "Ollama is running". i have a setup where example.com is hosting an ollama web frontend and the reverseproxy proxypasses example.com/api to the ollama service. the result is, that the ollama app refuses the host, but it could use it

I have the same problem

I hacked this together for myself. If you temporarily reverse proxy both the root path / and the ollama path /ollama/ to your Ollama instance you can convince the app to accept the settings. Then restore the reverse proxying to openwebui and it will continue working.


[tool_crash] FileSystemException: Cannot create link, OS Error: Incorrect function., errno = 1

Command
flutter pub get

Steps to Reproduce

...
...
...

Logs
FileSystemException: Cannot create link, OS Error: Incorrect function.
, errno = 1
#0      _Link.throwIfError (dart:io/link.dart:254:7)
#1      _Link.createSync (dart:io/link.dart:182:5)
#2      ForwardingLink.createSync (package:file/src/forwarding/forwarding_link.dart:20:16)
#3      ForwardingLink.createSync (package:file/src/forwarding/forwarding_link.dart:20:16)
#4      _createPlatformPluginSymlinks (package:flutter_tools/src/flutter_plugins.dart:1057:12)
#5      createPluginSymlinks (package:flutter_tools/src/flutter_plugins.dart:987:5)
#6      refreshPluginsList (package:flutter_tools/src/flutter_plugins.dart:1089:5)
<asynchronous suspension>
#7      FlutterProject.ensureReadyForPlatformSpecificTooling (package:flutter_tools/src/project.dart:357:5)
<asynchronous suspension>
#8      PackagesGetCommand.runCommand (package:flutter_tools/src/commands/packages.dart:179:5)
<asynchronous suspension>
#9      FlutterCommand.run.<anonymous closure> (package:flutter_tools/src/runner/flutter_command.dart:1257:27)
<asynchronous suspension>
#10     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
<asynchronous suspension>
#11     CommandRunner.runCommand (package:args/command_runner.dart:209:13)
<asynchronous suspension>
#12     FlutterCommandRunner.runCommand.<anonymous closure> (package:flutter_tools/src/runner/flutter_command_runner.dart:283:9)
<asynchronous suspension>
#13     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
<asynchronous suspension>
#14     FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:229:5)
<asynchronous suspension>
#15     run.<anonymous closure>.<anonymous closure> (package:flutter_tools/runner.dart:64:9)

[✓] Flutter (Channel stable, 3.7.0, on Microsoft Windows [Version 10.0.22000.1455], locale en-US)
    • Flutter version 3.7.0 on channel stable at D:\src\flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision b06b8b2710 (4 days ago), 2023-01-23 16:55:55 -0800
    • Engine revision b24591ed32
    • Dart version 2.19.0
    • DevTools version 2.20.1

[✓] Windows Version (Installed version of Windows is version 10 or higher)

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.1)
    • Android SDK at C:\Users\isede\AppData\Local\Android\sdk
    • Platform android-33, build-tools 33.0.1
    • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)
    • All Android licenses accepted.

[✓] Chrome - develop for the web
    • Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe

[!] Visual Studio - develop for Windows (Visual Studio Community 2019 16.11.21)
    • Visual Studio at C:\Program Files (x86)\Microsoft Visual Studio\2019\Community
    • Visual Studio Community 2019 version 16.11.33027.164
    ✗ Visual Studio is missing necessary components. Please re-run the Visual Studio installer for the "Desktop development with C++" workload, and include these components:
        MSVC v142 - VS 2019 C++ x64/x86 build tools
         - If there are multiple build tool versions available, install the latest
        C++ CMake tools for Windows
        Windows 10 SDK

[✓] Android Studio (version 2021.3)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866)

[✓] VS Code (version 1.74.3)
    • VS Code at C:\Users\isede\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.58.0

[✓] Connected device (4 available)
    • sdk gphone64 x86 64 (mobile) • emulator-5554 • android-x64    • Android 13 (API 33) (emulator)
    • Windows (desktop)            • windows       • windows-x64    • Microsoft Windows [Version 10.0.22000.1455]
    • Chrome (web)                 • chrome        • web-javascript • Google Chrome 109.0.5414.119
    • Edge (web)                   • edge          • web-javascript • Microsoft Edge 109.0.1518.69

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 1 category.

Flutter Application Metadata
Type: app
Version: 0.1.0
Material: true
Android X: false
Module: false
Plugin: false
Android package: null
iOS bundle identifier: null
Creation channel: stable
Creation framework version: b06b8b2710955028a6b562f5aa6fe62941d6febf
Plugins
agora_rtc_engine-6.1.0
iris_event-2.0.0
permission_handler-10.2.0
permission_handler_android-10.2.0
permission_handler_apple-9.0.7
permission_handler_windows-0.1.2

Closing as duplicate of https://github.com/flutter/flutter/issues/66224


Addressing missing name for Guest order.

Refactor to set Customer Name using Billing address details for Guest orders.
Fixes: #3
Fixes: #4

Sorry it took so long, next time do not hesitate to ping me ;)

Will do. GH didn't close the #3 and #4 for some reason. You might want to manually close them.
https://help.github.com/articles/closing-issues-via-commit-messages/


error on instaling the repository

When i try to run the bat file everything goes perfectly, but after the downloading, it shows "access is denied"
I don't know what I am doing wrong.

I fixed it by downloading it manualy and replacing the files.


An example of a query that uses "Contains words" would be welcome.

I have not been able to use "Contains words" in a TFS query.  If I have two or more words in the value, the query fails.  Even when we know there are items with the specified words.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6859c579-3383-e4b7-366e-69f49a8bd1b6
Version Independent ID: 5b7af784-0698-b77a-342f-252986038e92
Content: Query by title, ID, or rich-text fields - Azure Boards
Content Source: docs/boards/queries/titles-ids-descriptions.md
Product: devops
Technology: devops-agile
GitHub Login: @KathrynEE
Microsoft Alias: kaelli

@CLIRResearch -- you may find an example in one of these:

Azure DevOps on Stack Overflow
Azure DevOps Support Bot

@CLIRResearch - Thank you for the feedback. I will update the topic to provide more examples. I suggest meanwhile that you use one word per clause and add more clauses with an OR operator as needed. If you enter "performance and guidance", for example, for the value, then the query will find those work items with the field that contains that specific phrase.


fix command line multisig bug. Maybe code need to be formated or perfected

Traceback (most recent call last):
File "/usr/local/bin/electron-cash", line 4, in
import('pkg_resources').run_script('Electron-Cash==2.9.3', 'electron-cash')
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 748, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 1517, in run_script
exec(code, namespace, namespace)
File "/usr/local/lib/python2.7/dist-packages/Electron_Cash-2.9.3-py2.7.egg/EGG-INFO/scripts/electron-cash", line 423, in
result = run_offline_command(config, config_options)
File "/usr/local/lib/python2.7/dist-packages/Electron_Cash-2.9.3-py2.7.egg/EGG-INFO/scripts/electron-cash", line 286, in run_offline_command
result = func(*args)
File "/usr/local/lib/python2.7/dist-packages/Electron_Cash-2.9.3-py2.7.egg/electrum/commands.py", line 86, in func_wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/Electron_Cash-2.9.3-py2.7.egg/electrum/commands.py", line 245, in signtransaction
self.wallet.sign_transaction(tx, password)
File "/usr/local/lib/python2.7/dist-packages/Electron_Cash-2.9.3-py2.7.egg/electrum/wallet.py", line 1104, in sign_transaction
k.sign_transaction(tx, password)
File "/usr/local/lib/python2.7/dist-packages/Electron_Cash-2.9.3-py2.7.egg/electrum/keystore.py", line 114, in sign_transaction
tx.sign(keypairs)
File "/usr/local/lib/python2.7/dist-packages/Electron_Cash-2.9.3-py2.7.egg/electrum/transaction.py", line 829, in sign
pre_hash = Hash(self.serialize_preimage(i).decode('hex'))
File "/usr/local/lib/python2.7/dist-packages/Electron_Cash-2.9.3-py2.7.egg/electrum/transaction.py", line 727, in serialize_preimage
amount = int_to_hex(txin['value'], 8)
KeyError: 'value'

Reviewing.  where did you copy this code from?

If merged, this fixes #169
It seems it might be best though to hold off as we are also looking into #166 which looks like it will need a similar fix. If true, this may be something we wish to consolidate in a different location or revise slightly to handle all cases known to us.

Its due to BIP143 tx signing, which requires the values of the input to be included. As kenman345 pointed out, #166 is related.
Since previous tx's are needed, I think the logical place for this fix would be in the Wallet class. For an online wallet, the wallet should have all previous tx. Wallet.sign_transaction() is called from many places - solution could be to add a check here to include the input value if not already present.
Offline wallets would be an issue, but this is covered by #166.

as you said, the transaction inputs need the previous transactions. This might be better suited for the transactions class and not wallet. Would be nice if this could solve both issues.

The value of the input should be inserted into the tx object but the problem is the transaction class has no knowledge of other tx, or of how to find them.

So....is this the best option then? If Transaction class wont know the information, then is this the only way to grab it?

PR #196 is an alternative for a fix of #169. One or the other should be merged, not both.

Superseded by #196


Default Logger middleware can cause OOMs

Hi all,
I have noticed that the naive use of org.http4s.server.middleware.Logger with logBody = true can easily lead to out-of-memory (OOM) issues when handling very large requests or responses due to stream copying and serialization to a string.
While this can be mitigated using org.http4s.server.middleware.EntityLimiter, but in cases where you handle large uploads or downloads, such as in file server scenarios, it becomes quite complex to apply different middlewares with different settings (like logBody = false) for specific endpoints.
I believe it would be beneficial to provide an additional parameter in the Logger middleware that limits the number of body bytes to be logged, with a reasonable default. In my experience, I’ve encountered several http4s backends that log the full request/response body, which is not always useful and can result in performance issues.
For reference, I implemented this in a hobby project using Pull to limit body size logging. You can see an example here: DebuggingDirectives.scala.
Thank you for considering this enhancement!

This seems like a good idea.
I would probably solve this problem a different way: partition routes that log the body and routes that don't and compose them with <+>, wrapping only one set of them in the middleware with logBody = true.


fix(app): make button cta text the same across odd and app

closes RQA-2010
Overview
Fixes the button for updating a deck fixture so the text is the same between ODD and app
Test Plan
Look at the location conflict modal when there is a fixture conflict on ODD and app and see that the confirm button text says "update deck" for both.
Changelog

update the button text to say update deck only for both cases, and remove the branching logic for the app button
fix test
remove now unused i18n string

Review requests
see test plan
Risk assessment
low

seems that both of them using confirm removal.
desktop
https://www.figma.com/file/nJBVk8a26wdls3ZnkgRmns/OCT-Release%3A-Desktop-[Deck-configuration-%26-Drop-tips]?node-id=1%3A384&mode=dev
odd
https://www.figma.com/file/eMAHM8g07dtTKr2RsFjkzB/OCT-Release%3A-Touchscreen-[Deck-configuration-%26-Drop-tips]?node-id=1%3A386&mode=dev

seems that both of them using confirm removal.

@koji thanks for this! @ecormany mentioned in the ticket that Update deck makes more sense.


feat: modulo operator

Summary of Changes
Add the modulo operator %. It follows Python's floor-based implementation and defines n % d as
$$
n - \left\lfloor\frac{n}{d}\right\rfloor \cdot d
$$

:tada: This PR is included in version 0.21.0 :tada:
The release is available on:

npm package (@latest dist-tag)
npm package (@latest dist-tag)
v0.21.0
GitHub release

Your semantic-release bot :package::rocket:


IMPROVE - Voice Over / Voice Control Accessibility

Describe the feature you want to improve

Go through the DashboardView and SettingsView and check for proper voice over support

Describe alternatives you've considered

also look at Voice Control

@montanabraswell said she was interested in this one, please comment below so I can officially assign you to this issue

Hi, Can you please assign me this issue ?

Hi, sorry just saw this now, if this is still available I’d like to take it. :)

I am assigning this to @montanabraswell Like I mentioned in my first comment. @narendrajagne Please find a different issue to work on

ok

@montanabraswell  Are you planning on updating this in the next few days? Hacktoberfest ends in about 1 week, for it to count as a PR, it needs to be merged or accepted by the 31st.
If you're not interested in updating this let me know, or remove yourself as an assignee on the issue, thank you!

Hi Mikaela,
Yes! Thanks for the response - I'm working on it today. Sorry, work has
been a bit crazy the last few weeks. Thanks for your patience!
On Tue, Oct 24, 2023 at 1:49 AM Mikaela Caron @.***>
wrote:

@montanabraswell https://github.com/montanabraswell Are you planning on
updating this in the next few days? Hacktoberfest ends in about 1 week, for
it to count as a PR, it needs to be merged or accepted by the 31st.
If you're not interested in updating this let me know, or remove yourself
as an assignee on the issue, thank you!
—
Reply to this email directly, view it on GitHub
https://github.com/mikaelacaron/Basic-Car-Maintenance/issues/51#issuecomment-1776573863,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AF3AFOLOUFWYTEKNFSSOTXDYA5JGVAVCNFSM6AAAAAA5QQNSFGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZWGU3TGOBWGM
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
Montana Braswell
Certified SAFe 4 Scrum Master
[image: Certified SAFe® 4 Scrum Master]

@mikaelacaron Can I take this ticket?

@mikaelacaron Hi! For my first time utilizing the voice-over feature, I want to propose a solution for this ticket: I will add the .accessibilityLabel functionality to every Text/Link I see in both DashboardView and SettingsView, so the voice-over feature can be as clear as possible. What do you think?

@rbennum I think that's a good start! Are there other things that we should do to have better Voice Over and Voice Control support?

@mikaelacaron in case of Voice Control support, I suppose we can assign every link/button to the action name to be recognized by the voice control system. But with my current capability (I have never tried using and developing voice control), I think the scope is quite big so I want to separate it into each of their own, if that's okay with you.

When this ticket was originally written, the app was smaller.
How about you do full Voice Over and Voice Control support only for the DashboardView and that's it
@rbennum

@mikaelacaron I think I can manage that.

Great! Assigned you this issue!


Natspec

Codecov Report

Merging #98 (fe3802f) into master (7b23cc8) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master      #98   +/-   ##
=======================================
  Coverage   91.94%   91.94%
=======================================
  Files          39       39
  Lines        2457     2457
  Branches      365      365
=======================================
  Hits         2259     2259
  Misses        198      198

Impacted Files
Coverage Δ

contracts/PoolRestrictions.sol
100.00% <ø> (ø)

contracts/balancer-core/BPool.sol
98.11% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b23cc8...fe3802f. Read the comment docs.

Codecov Report

Merging #98 (fe3802f) into master (7b23cc8) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master      #98   +/-   ##
=======================================
  Coverage   91.94%   91.94%
=======================================
  Files          39       39
  Lines        2457     2457
  Branches      365      365
=======================================
  Hits         2259     2259
  Misses        198      198

Impacted Files
Coverage Δ

contracts/PoolRestrictions.sol
100.00% <ø> (ø)

contracts/balancer-core/BPool.sol
98.11% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b23cc8...fe3802f. Read the comment docs.


Ensure permissions on /etc/hosts stay intact

Fixes issue https://github.com/smdahlen/vagrant-hostmanager/issues/160

Thanks! This looks reasonable.


Asking about "import Comparative_models.CE as CE"

In the file "detection.py," I found "import Comparative_models.CE as CE" on line 466. How I can get this package? Moreover, I want to run your model based on the dataset  "MVTec". Would you please advise me on how I can reproduce the result? What the configuration file should look like and the steps to reproduce the result.

That first module is the context encoder used to generate comparative statistics for the paper. I wasn't planning on uploading that but I can do if you'd still like it?
So I ran the model on the leather subset of MVTec so you should be able to use the MTec Dataset class and load in the model that way - although when instantiating and using the model throughout I've referred to it as "leather", but you should be able to edit those sections and run it ok - the config file I used looks like (changing the noise_fn for simplex for that model):
{
  "img_size": [256,256],
  "Batch_Size": 1,
  "EPOCHS": 3000,
  "T": 1000,
  "base_channels": 128,
  "beta_schedule": "linear",
  "channel_mults": "",
  "loss-type": "l2",
  "loss_weight": "none",
  "train_start": true,
  "lr": 1e-4,
  "random_slice": true,
  "sample_distance": 600,
  "weight_decay": 0.0,
  "save_imgs":false,
  "save_vids":true,
  "dropout":0,
  "attention_resolutions":"16,8",
  "num_heads":2,
  "num_head_channels":-1,
  "noise_fn":"gauss",
  "dataset":"leather"
}

Thank you very much for your advice. Now I can train the model. By the way, Can you explain in detail how to evaluate the model ? I don't know how to use detection.py

Firstly, you must ensure the function you want to run is selected at the bottom of the file and linked under the arg file provided.
Since you're running the model on MVTec, the anomalous_metric_calculation function will loop over the leather dataset and record various relevant metrics to a file. This is the primary function used for evaluation.
Alternatively, if you want to reproduce the graphs for example, then the roc_data and graph_data functions sort this however they would need to be adapted for your dataset and remove redundant comparative models.
Let me know whether this helps

Closing due to no responses

Hello!
Recently，I'm learning the code. I see "import Comparative_models.CE as CE" in some modules. And I can't find the module in the AnoDDPM repo. So I want to know how to get this module?
Looking forward to your reply! Thank you!

Thank you very much for your advice. Now I can train the model. By the way, Can you explain in detail how to evaluate the model ? I don't know how to use detection.py

Hello, I am also facing the issue with "import Comparitive_models.CE as CE". I checked this thread but couldn't find the uploaded Comparitive_models. Have you found a solution to it?

In the file "detection.py," I found "import Comparative_models.CE as CE" on line 466. How I can get this package? Moreover, I want to run your model based on the dataset "MVTec". Would you please advise me on how I can reproduce the result? What the configuration file should look like and the steps to reproduce the result.

Import Comparative_ Models CE as CE, how did you solve it


tool to convert arbitrary svg to IconSource json

Do you already have a convenient way to take an svg and convert it into the IconThemeSource format that is ready to be used with the Icon component?

@ronkeiser For this case you can copy the template-builder!
Just copy your svg files to themes/default, update package.json (name, author etc) and run pnpm i && pnpm publish --no-git-checks and you can use your custom icons like you usually would with this library.
With the build.js file you can further configure how your sources will be treated (e.g where to collect them from or exclude/extend Attributes for the output)
Hope this helps!! (reopen if this wasn't what you were looking for)

Perfect!


fix flakey race in ruby tests

cc @muxi, this should fix the flake seen earlier

cc @murgatroid99 actually there is one more change I'd like to make before this is ready

actually sorry, I was looking for a way to avoid the sleep 5 but I don't see any way to do this.
I think this is ready for review

Thanks for the fix Alex.

No significant performance differences

No significant performance differences

tests are green


JSON spec tests for roles and policies

add spec tests for roles and policies

partially addressed in #88

completed in #90


[ActionBar] offscreen actions are cloned in more popover which results into action items with duplicate id such as coral-popover

Expected Behaviour : There shouldn't be action items such as button and popover with duplicate id.
Actual behaviour : There are  actions items such as button and popover with duplicate id.
Reproduce Scenario (including but not limited to)
When actions with popover goes offscreen, buttons and popovers from these actions are cloned into more popover [0], [1] which results into duplicate items with same id.
[0] : https://github.com/adobe/coral-spectrum/blob/master/coral-component-actionbar/src/templates/overlayContent.html#L20
[1] : https://github.com/adobe/coral-spectrum/blob/master/coral-component-actionbar/src/scripts/BaseActionBarContainer.js#L188
Steps to Reproduce
Browser name/version/os (ie Chrome Version 62.0.3202.94 (Official Build) (64-bit) MacOS)
Coral Spectrum version
4.8.5
Sample Code that illustrates the problem (use the Playground if possible)
See in the below playground, the relate action inside more popover doesn't open linked popover on click because 2 popover s with same id exists and the display call goes to first popover which is offscreen.
link_to_reproduce
Screenshots (if applicable)

@icaraps  Could you please help us in fixing this issue. this is root cause of CQ-4299022.

@indra2gurjar you can iterate over all elements with ids and remove the hidden elements with ids, and if the elements are visible again you can restore the ids.


index.d.ts typings for typescript users

If I submitted a PR with an index.d.ts addition that would add typings so that typescript users could use this library more easily, would it be accepted?

You can always submit a PR to DefinitelyTyped/DefinitelyTyped

I did actually, that was my first accepted PR on any project. I was just offering it here too.


Design-Simple Queue

[ ] Two files in /lib/:

[ ] simple_queue.cpp

[ ] simple_queue.hpp

[ ] Create a directory simple_queue_cpp in examples

[ ] Create a main.cpp

[ ] Create a makeFile


Script File Throwing Error on Line 85

Hi there,
AWESOME STUFF first of all.
So in codepen, everything is working just great (http://codepen.io/andandandandrew/pen/XmKEMK). I'm importing the theater.js file then typing the plugin scripty part in the JS panel, which is tantamount to an inline script if I'm correct.
So where I'm trying to use this on a site I'm working on, I have the theater.js script dynamically loading using PHP (it's only used on one page) loading using src="" after the footer, immediately followed by another inline script with the following code in it, which is also php'd in as an inline script:

I am getting a console error (screenshot attached) on line 85 of the script file: Uncaught TypeError: Cannot set property 'innerHTML' of null

Any idea what's going on here? It's being used on the homepage of a WordPress Site and is not typing anything, even though the markup is the same as in the pen (id to JS it is anyways). This is where it should be typing, but it's not:

Any ideas? Is it the PHP loading that's doing it? I've copied the script exactly from the repo—there's not something in the actual script file I need to update, right?
Thanks for the help.

Thanks @andandandandrew, glad you like it ;)
As it stands, it looks like it's trying to set the innerHTML of a null variable - which means that the DOM element couldn't be found. There are two possible causes:

TheaterJS tries to access the DOM element before the DOM is ready.

To fix this issue, you can whether wrap the code in a "DOM ready" listener: document.addEventListener('DOMContentLoaded', function() { /* the code goes here */ }, false); or put your scripts at the end of the <body> tag.

There are no element matching the actor's selector.

Make sure that the selector passed as the actor's "voice" is correct, e.g theater.describe('actorName', {/* props */}, '#some-selector'). You can check this by opening your browser's console and execute document.querySelector('#some-selector').

Let me know if that do help :)

OK, so I am a super JS idiot, so please bare with me. I added the DOM ready listener to theater.js by prepending "document.addEventListener('DOMContentLoaded', function() {" before the start of the file's code, and appended "}, false);" after the last line in theater.js.
That solved the inner HTML null issue. But now, I have another issue.
Now I'm getting this message: Uncaught ReferenceError: TheaterJS is not defined in the console.
Here's a screenshot that proves the actor and correct selector are being used. Any thoughts? Did I do it wrong. There's a very good chance I did.

Thanks for your help my developer brother in arms; you are a rockstar!

Actually the "DOM ready listener" should wrap the code that is actually manipulating the DOM: your code (but not the TheaterJS library).
That been said, I can see that the scripts are already placed at the end of the <body> so you shouldn't have to add the listener. Give it a try anyway and if that still does not solve your issue, my next bet is that the html content is loaded at a later time.
Any chance I could access to a version of this page? If not, try the following: when you're on the page, in your browser, use the browser's functionality to look at the source code (not from the inspector). Make sure you can find the <span id="typeit" /> in there.

I'll recreate the page on my personal site tonight and send you the link. It will still function the exact same way that this one is supposed to. Thanks for all the help, it's awesome!

Hi there. I just downloaded the webpage and threw it up on my personal site's server. Pleases excuse the trash that is the markup in there. I really appreciate you taking a look at the code.
http://and-rew.com/lift

I couldn't load the page correctly but looking at the source code, I found that the span's id is typeit lift-is which is not matched by the #typeit selector.

Looks like a mistake since an element can't have two ids and spaces are not allowed within an id name. So to get it right you'll have to: remove the "addEventListener" code from theater-min.js and use a proper id. Give it a try and let me know ;)

IT WORKS!!! Thanks so so so much. I think what it was is that I had 2 IDs in the inline script part, why? I HAVE NO IDEA! I feel dumb, but am literally dancing around about it now!

haha really glad to help and thanks for being so enthusiastic :)


Mobile Safari - Failed to play local media

Describe the bug
When use in mobile Safari the audio of the very first call is not working. After call starts the permission dialog is shown. After allowing to use mic audio does not work. The call itself is in progress and probably works ok. Second call will not produce a permission dialog and audio works ok. After the page reload the described case repeats again.
Everything works ok in desktop Safari. So it is only a mobile Safari issue.
Logs

To Reproduce (if possible)
I am using SimpleUser with registration on the company server  so could not provide the full code sample. Here is a pseudocode:
const su = new Web.SimpleUser(server.url,  {
 aor: params.aor,
 userAgentOptions: {
   authorizationUsername: params.identity,
   authorizationPassword: params.credentials
 },
 media: {
   local: { video: mediaElements.localVideoElement },
   remote: { video: mediaElements.remoteVideoElement }
 }
})
su.delegate = {
 onRegistered: () => su.call(...)
}
await su.connect()
await su.register(...)

The elements:
 <div class="videos-group">
    <video class="remote-video" id="remoteVideo" ref="remoteVideoComponent"></video>
    <video class="local-video" id="localVideo" muted="muted" ref="localVideoComponent"></video>
 </div>

Expected behavior
No error should be logged
Observed behavior
There are errors related to audio permissions
Environment Information

sipjs version 0.21.2
Mobile Safari, iOS 16

Ok, playsinline attribute solves the problem. Thank you.


🐛 figmabroker: fix prettier.resolveConfig.sync

this function was removed in prettier 3.0. prettier.resolveConfig remains as an async function so the code probably needs to be updated to be async (unless we can get rid of it)

prettier.format was also made async
I have commented out the usage of these functions in scripts\figma-broker\functions\file.js. This is used to create packages\eds-icons\src\data.ts, so for the time being, that file can be formatted by opening it in vscode and the autoformatter will take care of it on save.

Alternatively we can add packages/eds-icons/src/data.ts to .pettierignore. I don't see why not 🤷♂


Clear the Course Nbr value when you change the Subject

This is related to #6 . Closing it and adding it there.


[5.5] Maintain original order of collection items with equal values when using sortBy()

This PR makes the collection sortBy() method perform a stable sort, such that it preserves the original order of collection items which have identical values.
The problem
sortBy() relies on asort, which, as @sisve pointed out during this discussion in laravel/internals, includes this warning:

If two members compare as equal, their relative order in the sorted array is undefined.

So, for example, if you run sortBy('letter') on this collection:
collect([
    ['letter' => 'b'],
    ['letter' => 'a'],
    ['letter' => 'a'],
]);

...the resulting collection swaps the order of the two 'letter' => 'a' items:
Illuminate\Support\Collection {#54
  #items: array:3 [
    2 => array:1 [
      "letter" => "a"
    ]
    1 => array:1 [
      "letter" => "a"
    ]
    0 => array:1 [
      "letter" => "b"
    ]
  ]
}

This behavior varies depending on the nature of the collection—sometimes the original order will be maintained, sometimes it won't.
Why it matters
This becomes a particular problem when trying to sort a collection by multiple criteria. One way to do so is to chain sortBy() clauses in reverse order, i.e. in order to sort by "foo then bar then baz", you should be able to call:
$collection->sortBy('baz')->sortBy('bar')->sortBy('foo')

...but, because the original position of duplicate values isn't necessarily maintained between each call to sortBy(), this will sometimes result in a correct multi-sort, but not always. The results are unpredictable.
The solution
The simplest solution I've found involves using the amusingly-named Schwartzian Transform, which basically involves converting each value in the array you're sorting (['b', 'a', 'a']) into an array that includes its index ([['b', 0], ['a', 1], ['a', 2]])—which asort will handle correctly—and then stripping out those added keys once the sorting is done. Luckily, the way the sortBy() method is written, this is easy—we only need to add the keys during the first loop.
All in all, an additional 8 characters of code...and a much, much longer description. 😄

Reverted in https://github.com/laravel/framework/pull/21255


Team 12 VnV Report Review - Lack of Input and Output states for unit tests

Hey, solid VnV Report! One thing I noticed was that in your unit tests (Section 5), the tests for the methods of each module do not include input and output states. I feel that including them can possibly provide more insight when conducting an evaluation of each software module.
This issue was created by Team 12 as part of VnV Report Review. Tagging marking TA as recommended @samm82
[Review-Team12]

Completed as part of the final document update


Japanese GUI font-family

I suggest about Japanese GUI font-family. (not editor font family)
Current vscode specified "Meiryo" in Japanese font-family, but normally Windows 10 using "Yu Gothic UI" as Japanese system font.
We should add Yu Gothic UI to vscode font-family because Yu Gothic UI has a better character width and rendering than Meiryo.
Current (workbench.main.css)
.windows:lang(ja) {
    font-family: Segoe WPC, Segoe UI, Meiryo, sans-serif
}

Suggestion (workbench.main.css)
.windows:lang(ja) {
    font-family: Segoe WPC, Segoe UI, Yu Gothic UI, Meiryo, sans-serif
}

@bpasero I think this is something set in the workbench.

@harujii wanna do a PR for this change

@bpasero I Created PR #79735


Do notation

This is a proposal to start a conversation about introducing a do-notation, using Maybe as an example.
See the docs and tests for details 😎
Can't wait for your feedback Dr Haskell @pzel and Dr Rust @arkgil 😊

@studzien looks great! My only question is about the return values. When we unwrap using an arrow in a block as the last expression, it looks like the unwrapped value is a return value of the block. However if the last expression is not an arrow, it needs to be an explicit maybe.
What do you think?

Okay, now I see why we need to return a maybe from a block - otherwise, this wouldn't make much sense :D
BTW, the macro fails if it wraps a single arrow match since you handle them only when they are in blocks. In other cases the expression is unquoted and fails with undefined <-/2 function:
Maybe.with do
5 <- Maybe.just(5)
end

I don't want to introduce macros, at least not for now ;)


arch-riscv: Move fault handler addr logic to ISA

mtvec.mode is extended in the new riscv proposal, like fast interrupt. This change moves that part from Fault class to ISA class for extendable.
Ref: https://github.com/riscv/riscv-fast-interrupt
Change-Id: Ifcfd6f4ce011149b5982dabfd2421f48c9a10b75

CC @rogerchang23424

Wait for #522 merged to avoid conflict


Broadcast lastTargetedRoll information to all clients.

So that a GM could use a macro like:
let sel = canvas.tokens.controlled
if (sel.length == 2) {
  let a1 = sel[0].actor
  let a2 = sel[1].actor
  let d1 = GURPS.lastTargetedRoll[a1.id]
  let d2 = GURPS.lastTargetedRoll[a2.id]
  if (d1 && d2) {
    if (d1.margin > 0 & d1.margin > d2.margin)
      ui.notifications.info(a1.name + ' wins')
    if (d2.margin > 0 & d2.margin > d1.margin)
      ui.notifications.info(a2.name + ' wins')
  }
}

To do a quick contest between characters, etc.

You read my mind! That's always been the plan.
One thing that I don't like about lastTargetedRoll is that it is tied to the actor only. Unlinked tokens from the same actor overwrite each other.

Done.
GURPS.lastTargetedRoll is local to the current client and is NOT broadcast to all clients.
GURPS.lastTargetedRolls is an object of last targeted roll chatdatas, indexed by both the actor ID and the token ID.    This information is broadcast to all clients.


Do not open issues -- start a discussion instead!

Going forwards, issues are only for things which I have decided to work on.
Please open a discussion instead, in an appropriate category. If you have found a bug, or suggested a feature which is accepted, I'll turn the discussion into an issue.

Closing, as the issue template now makes this clear.